Passenger safety circuitry and passenger safety method

Abstract

The present disclosure generally pertains to passenger safety circuitry for warning an operator of a vehicle with an autonomous driving function, the circuitry being configured to: determine an operator readiness level based on an operator foot position; determine whether an operator warning is to be issued; and based on the determined operator readiness level, issue the operator warning.

Description

[0001] PASSENGER SAFETY CIRCUITRY AND PASSENGER SAFETY

[0002] METHOD

[0003] TECHNICAL FIELD

[0004] The present disclosure generally pertains to passenger safety circuitry and a passenger safety method for an automated or partially vehicle.

[0005] TECHNICAL BACKGROUND

[0006] Some road segments may provide the capability of driving under full or partial (or conditional) automation. There may be different types of road segments, such as with or without a stop lane (hard shoulder) or evacuation spot (e.g., in a tunnel, metropolitan motorway, narrow bridge, and the like). Moreover, a usage of a minimal-risk maneuver (MRM) or a risk-mitigation function (RMF) in certain situations may cause a higher risk of a collision and thus, MRMs or RMFs may not always be useful.

[0007] Hence, a driver may be required to adaptively interact with the automation system and may be required to take over to manual driving.

[0008] It is generally known to issue a notification to the driver that such a transition is required.

[0009] Although there exist techniques for warning a driver of a vehicle, it is generally desirable to provide passenger safety circuitry and a passenger safety method.

[0010] SUMMARY

[0011] According to a first aspect, the disclosure provides passenger safety circuitry for warning an operator of a vehicle with an autonomous driving function, the circuitry being configured to: determine an operator readiness level based on an operator foot position; determine whether an operator warning is to be issued; and based on the determined operator readiness level, issue the operator warning.

[0012] According to a second aspect, the disclosure provides a passenger safety method for warning an operator of a vehicle with an autonomous driving function, the method comprising: determining an operator readiness level based on an operator foot position; determining whether an operator warning is to be issued; and based on the determined operator readiness level, issuing the operator warning.

[0013] Further aspects are set forth in the dependent claims, the drawings, and the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Embodiments are explained by way of example with respect to the accompanying drawings, in which:

[0015] Fig. 1 depicts a passenger safety method according to the present disclosure;

[0016] Fig. 2 depicts a further passenger safety method according to the present disclosure in which encrypted data is generated;

[0017] Fig. 3 depicts a block diagram of a method for determining an additional risk factor based on an evaluation of an operator’s emotional status;

[0018] Fig. 4 depicts an illustrational diagram representing how a positive loop may be created for the operator;

[0019] Fig. 5 depicts a timeline according to the present disclosure showing which action is taken at different points of time before a critical point (overall time budget) is reached before safe continuation of the trip cannot be guaranteed anymore;

[0020] Fig. 6 depicts an interior of a vehicle in which circuitry according to the present disclosure is used;

[0021] Fig. 7 depicts a flow chart of a vehicle / passenger safety method according to the present disclosure in which the notification escalation is depicted;

[0022] Fig. 8 depicts an embodiment of a passenger safety method according to the present disclosure;

[0023] Fig. 9 depicts an embodiment of a passenger safety method according to the present disclosure in which databases are updated based on the operator’s reaction;

[0024] Fig. 10 depicts an occupant monitoring system according to the present disclosure;

[0025] Fig. 11 depicts a method for generating body feature information according to the present disclosure;

[0026] Fig. 12 depicts an illustrational result of a method according to Fig. 11;

[0027] Fig. 13 depicts a skeleton in a depth image and a result of segmentation in a confidence image;

[0028] Fig. 14 depicts a depth image, a confidence image, and an image in which distances between a driver and a seat are depicted;

[0029] Fig. 15 depicts a method for ToF circuitry and / or for a vehicle safety system according to the present disclosure in a block diagram; Fig. 16 depicts a method for a vehicle safety system according to the present disclosure in a block diagram;

[0030] Fig. 17 depicts a flow chart of a method for deciding whether a more precise occupant tracking mode is applied; and

[0031] Fig. 18 depicts a flow chart of a method that is carried out in the more precise occupant tracking mode.

[0032] DETAILED DESCRIPTION OF EMBODIMENTS

[0033] Before a detailed description of the embodiments starting with Fig. 1 is given, general explanations are made.

[0034] As mentioned in the outset, vehicle safety systems are generally known. However, if a safety system reacts improperly, it may be desirable to suppress or to re-configure the employment of the safety system. For example, in the case of an airbag safety system which typically provides increased safety in critical situations, it may also cause damage or injury if improperly deployed, as commonly known. To maximize the airbag benefit while minimizing the undesired risk, it has been recognized that it may be desirable that a planned deployment of an airbag may be suppressed on re-configured. For example, if a head of a passenger is tilted, a deployment strategy may be different than when the passenger is facing straight. Depending on whether a driver intervenes during automated driving, a deployment strategy can also be different from when a driver does not intervene. Therefore, it has been recognized that a determination of whether the driver is about to intervene or is at least ready to intervene may have an influence on other safety systems or other road users in proximity to the vehicle and furthermore may disturb fluent traffic flow. In the example of the airbag system, the present disclosure may provide a possibility to enable the airbag deployment to perform in accordance with smart detection of a driver / passenger based on a precise posture, behavior in a pre-classified manner to so that the airbag can inflate based on the smart detection.

[0035] Moreover, it has been recognized that it may be desirable to determine when a driver (or operator - in this specification the terms “driver” and “operator” are used interchangeably, in some instances and embodiments, and these terms should be understood as synonymous as long as not indicated otherwise in concrete examples.) becomes distracted or preoccupied during an automated driving although the driver is expected to be capable of intervening at any given time. This may be particularly appropriate with level 3 automation. In some instances, alarms or alerts can be used to indicate to a driver to be ready to intervene. It has been recognized that it may be desirable to suppress a false alarm or alarm issued after the driver has already demonstrated that the driver intends to intervene since repeated late and false alarms may lead the driver to ignore the alarms which increases a risk for an accident, for example. On the other hand, fewer false alarms may lead to a higher or improved motivation for the driver to intervene when a notification is issued rather than reaching a situation with a high degree of criticality and relying on an alarm to indicate that level of criticality. Also, some events may not require direct intervention, but may only require that the driver is in principle ready to intervene or take action, i.e., to refocus their gaze, take control of the steering wheel, be ready to press the brake pedal, and the like.

[0036] It has further been recognized that unnecessary alerts may lead the driver to ignore the alerts. Although the warnings may be necessary, they may be badly timed if the driver has already made an intended decision to intervene, such that the driver may start ignoring them. For example, if warning is issued often when the driver had already made decision to intervene, it may be considered as badly timed and also as unnecessary. Therefore, it may be desirable to reduce false or unnecessary alerts by detecting as early as possible the decision taken by the driver to intervene (or at least to detect the intention to make such a decision or to detect that the driver is ready to intervene). An early observable aspect, which may indicate that the driver intends or makes a decision, e.g. to intervene, is the start of the transfer of the driver’s foot from its home rest position while in automated mode to a position able to actuate the pedals when maneuvering manually. It has been recognized that this kind of motion rarely occurs unless a concrete decision is taken with intent. This is due the relatively large force required by the driver to lift and transfer their heavy leg to enable the foot to reach the acting point, for example, the pedal.

[0037] For example, MRM or RMF may be used as a fallback function with an automated driving function. Such functions may bring the vehicle to a stop whenever the system determines that the automated driving can no longer continue its planned travel safely and the driver is not capable to take over the dynamic driving control task as expected by the system, thus serving as last resort or safeguard while the vehicle is engaged in an automated driving mode.

[0038] However, overreliance on MRM or RFM may need to be prevented and it may only be used to cope in situations when a real emergency situation occurs, because otherwise, it might demotivate the driver from intervening, thereby deviating from its original purpose of having the MRM functions as a final safeguard system. Additionally, the easy use of MRM or RFM may create a false sense of safety or security for the driver. In some instances, the system constantly estimates the time budget to cope with an incoming / approaching certain situation and whenever the remaining time approaches a marginal limit, a countermeasure may be initiated in advance accordingly. With enough time budget left, the driver may be notified of the situation, but this does not necessarily mean that the driver will follow and initiate preparing themselves for the transition to manual driving. To the contrary, this time budget may still be used by the driver to a point where the system aborts the planned transition to manual driving and initiates the use of MRM or RMF, e.g. when reaching a critical irreversible situation. However, MRM and RMF are not supposed to be functions to support lazy drivers who consume the time budget before initiating / resuming the dynamic driving task. On the other hand, the driver may be required to take their best efforts to abort any non-driving related activity (NDRA) which may cause the consumption of the remaining time budget after the system issues a pre-notification and / or notification for a request to intervene. In some instances, the system can only estimate the time required for the driver to terminate a possible non-driving related task, e.g. based on a periodic or constantly monitored steady state of the driver and their positioning. The system may not be able to judge whether there is a possibility for the driver to be prepared before the remaining time budget reaches a critical decision point for issuing a MRM or RFM, if the driver is non- responsive to the initial notification, because, e.g., the driver is possible in another task. In some instances, such a situation then triggers a warning and further escalates to an alarm, thereafter triggering the MRM / RMF function. The term warning and alarm may be used interchangeably, but in some instances the warning initially reminds the driver that the time budget is being consumed and that the time budget for solving an approaching situation decreases. The alarm, which may be issued as a further alert, may have the purpose to remind the driver that a hard (or soft) stop will be automatically initiated soon by the automated system. Thus, the alarm may serve as reminder to the driver to prepare and may have the purpose to avoid that the driver and passenger(s) are surprised, e.g., by an unexpected activation of the MRM / RFM function. Such a surprising activation of the MRM / RFM function may cause risks, such as being hit by surrounding other traffic or an abrupt preventive maneuver in the course of accomplishing an automated MRM maneuver. Such an abrupt preventive maneuver may cause a risk that surrounding other road users / driver could be momentarily inattentive, which, in turn may cause a conflict with the own vehicle initiating an MRM maneuver. Generally, MRM and RMF could always become a source of disturbing a smooth running traffic flow, wherein such MRM and RMF are not expected by most of drivers in the vicinity of a vehicle. As it is expected that the number of automated vehicles on the road increases, such occurrences of unnecessary MRM activation and its resultant deceleration and stop in a busy traffic situation may even become a social problem, since over-use of MRM or RMF may have a negative impact on traffic, e.g. by causing disturbance to a smooth operating traffic flow, which in worst case may escalate into different types of critical situations, such as rear end collision, in the case that there is an incoming distracted driver being relaxed within a smooth traffic flow and be taken by surprise. Such inattentive driver may fail to cope a sudden deceleration of the front vehicle which could be urgently deceleration as part of the MRM, wherein the MRM may pay little attention to rear incoming other vehicles, which then, in turn, may cause further collisions or MRM actions. There is an environmental impact for MRM or RMF disturbing efficient traffic flow as kinetic energy is lost under braking and energy is needed to restore a vehicle to an appropriate speed when compared to smooth traffic flow. In examples a time budget may be an amount of time needed or a system-determined amount of time or a variable amount of time depending on changing circumstances influenced by system inputs.

[0039] Automated vehicles may (e.g., by law) have to be fully equipped with multilayered safety functions to keep the vehicle safe whenever the vehicle encounters a situation requiring appropriate driver intervention. Thus, in some instances, the driver may rely on such multilayered safety functions (e.g. too much), which may cause that the driver does not always resume as expected by the system. Moreover, known concepts may cause that the driver gradually loses the motivation to be constantly prepared to take an action as the multilayered safeguard like MRM or RMF provide the associated safety, which contradictorily affects the driver’s mind, removes out the perception to imminent risk, etc. This may result in the long term in decreasing driver sensory sensitivity, may fade out important and multiple knowledge accumulated in the driver’s memory that forms appropriate situation awareness. This, in turn, may degrade the driver’s capabilities when coping a situation at emergency, or the capability to proper allocate working memory in order to be actively able to cope the situation. This degradation of the driver’s capabilities may result, for example, in failure of the Object and Event Detection and Response, due to the overflow in the task beyond the driver’s working memory, which is otherwise allocated or gives priority to somewhere else instead of giving priority and allocate the working memory for coping for a possible risk.

[0040] Thus, it has further been recognized that it may be desirable to increase the driver’ s / operator’s situational awareness under the use of automation, e.g. by appropriate use of warnings, but it has its limitation. For example, unnecessary warnings may be ignored or missed by the operator which may lead to the case that important or critical warnings are ignored. It has been recognized that psychological cognitive information filtering may progress and occur when repeated false warning is unnecessarily issued in a situation where the driver had already appropriately taken decision and initiating to intervene following a requested intervention by the system. Thus, the warning may become rather a disturbance to the driver attention, which, in turn, causes that the driver may ignore or pay less attention to such warnings. Hence, in order to avoid that such an information filtering develops in the driver sensory and cognitive information path, timing and appropriateness of warning and alert plays an essential role in some embodiments as describe further in this disclosure. In some embodiments, when it has been determined that the operator has exhibited the intention to intervene in a timely manner, the system may take best efforts to suppress an unnecessary warning or alert unless a further instruction alert may be necessary, because an annoying alert may not be helpful, but instead may reduce the information processing resource of the driver, which may have a negative affect from safety perspective. In general, it is known that an excessive information capture of human sensory organs leads to the following development of information filtering ability. The human brain and human sensory organs may allocate higher sensitivity to items of interest and may disregard information of low interest, wherein this development is built upon experience. The inappropriate warning forms an example, which may cause such a development of information filtering ability.

[0041] In some embodiments, situational awareness means the combined knowledge of information that is collected over a time period. This knowledge can fade while driving, for example, based on the subjective risk level, but it can also be partly restored before doing a task, such as dealing with an emergency situation mostly without thinking about factors that may influence the outcome. For instance, a driver who is aware that the road is icy may brake gently and not suddenly to prevent the car from skidding and losing control, but a driver who is not paying enough attention may miss the warning sign on the road and do otherwise. Additionally, the choice of how much to brake depends on whether the driver knows that the vehicle lacks anti- slippery tires that could otherwise help deal with the frosty weather condition on the day of driving. This example illustrates that the driver gains situational awareness from various minor details, but their awareness may decline when they rely on automation, as they tend to lose focus during the drive and not retain all the relevant information for quickly and effectively restoring the appropriate situational awareness for resuming manual control under the current situation. Also under automation, a sensory window may decrease since an operator may rely on the functionality of the automatic system getting less attentive to perceive, digest and capture the relevant information. The sensory window may be widened when there is an imminent or "visually" perceivable risk to take care of. For instance, a traffic sign installed along the road is often an old fashion artificial method to remind the driver to keep attention as the road situation ahead deserves some care or specific attention. However, under automation, the driver sensory window may be narrowed and if this progresses beyond a certain limit even warnings may be ignored. Another cause for ignoring a warning may be, for example, that the operator is busy with another task. Hence, a decreased sensory window may lead to a decreased situational awareness at the point of time where a decision is required.

[0042] Also, the following non-limiting examples which typically require situational awareness may be missed or ignored by the operator since they are persistently exposed to a safe condition caused by automation which may gradually decrease the operator’s attention: a road sign (temporary or permanent), road temperature, side wind, weather conditions, vehicle status knowledge like transport of dangerous materials which may require to drive carefully, heavy load, reaching a dangerous / curvy road, region requiring snow tires, school zone, road under construction, and the like. It has been recognized that the automated driving system should interact with the driver in order to minimize the decrease of their situational awareness and / or to properly determine whether the driver is prepared or ready for taking the wheel whenever system becomes incapable of continuing a safe drive and requires to handover the control to the driver.

[0043] Hence, under automation, there may be a bottle neck in brain capacity of the operator leading to the ignoring of important warnings, because automation may leave increased room for interruptions that may be independent from the driving task (e.g., mind-wandering, distraction, using a mobile phone, or the like). Moreover, if the operator tries to do multiple tasks in parallel, a decision error may occur which may have critical consequences in traffic. For example, in order to take over and be prepared to resume manual driving, the driver may need to regain situational awareness and focus and increase their sensitivity in order to capture safety relevant information. But this does not occur instantly. Also, mind-wandering needs to be stopped or prevented in advance, and the maximum working memory may be reallocated in advance of reaching critical transition limit points. If the driver has no chance to do that, they may take a reflex or instinctive action with decreased consideration on the effect. When a driver oversteers or brakes hard, they may lose control of the vehicle, but they may not notice their mistake because the vehicle has automated features like ABS or ESC function, which automatically correct the driver's errors that could happen often but go unnoticed. However, this kind of impulsive behavior is not the only error caused by reduced situational awareness when using automation, and therefore, methods to improve or restore situational awareness may be necessary to prevent such behavior, which may be achievable according to the present disclosure.

[0044] In known automated vehicles, an artificial risk may be generated in order to condition the driver to react to the notification and warnings of the system. One way that the MRM function can be used to stop the driver from using the automation again if the driver repeatedly ignores the vehicle instruction to take over manual driving and deliberately neglects the notification and warning, is to force the MRM to start and have the vehicle stop temporarily as a penalty and punishment, not because of an accidental emergency situation but because of the driver's preference to continue a secondary task. However, it has been recognized that such artificial risks like bringing a vehicle to stand-by (= a term equivalent to "stop" in some instances within legal documents) may cause severe consequences, even when this stop is performed within an evacuation designed lane receptive to an emergency vehicle to come to stop. For example, if due to the above-mentioned artificial risk caused by activation of MRM or RMF for inducing artificial risk feeling to the driver as penalties in location not suitable to stop or location with low visibility, an unnecessary collision by another rear approaching vehicle may occur. Moreover, the road condition may have its assigned emergency lane already occupied by vehicles that failed to transition, or the local weather situation could be bad enough to obscure the visibility of other halted vehicles ahead. On the other hand, if the artificial risk is generated always in safe conditions, the driver may recognize it to be in safe conditions and not react to the warnings. Also, humans may have a tendency to react to a risk at the latest possible point of time, e.g., for avoiding use of unnecessary energy if it turns out that the risk is not “subjectively” high or imminent to occur. Moreover, negative conditioning like punishment can be considered to be not as effective as positive conditioning, such that psychologically it would be more useful to praise the driver instead of punishing the driver. Hence, it has been recognized that it may be desirable to provide countermeasures which create a positive reaction loop, such as praising the operator for pro-active behavior. Therefore, it may be better to provide incentives that encourage the driver to act proactively, such as complimenting the operator for good behavior. This driver mind conditioning concept is supported by psychological research that shows that positive feedback reinforces the neural connections of correct decisions, while negative penalties do not help form a desired decision loop in the operator’s mind. Negative penalties do not improve the decision making neural network, but only the penalty avoiding one, which does not help in enhancing safety.

[0045] Moreover, according to the present disclosure, an eco-system for self-motivation of the driver may be generated, e.g., by creating a reward for the driver. In some embodiments, this is combined with a visualizing for or of possible risk(s), e.g. when a recording system is provided. This visualization, in turn, boosts sensitivity, re-allocates working memory, and serves in maximizing the situation awareness that otherwise may just decrement with prolonged usage of automation along the trip. Although prolonged or repeated enforced stops may be able to provide a sense of penalty to the driver willing to reach destination sooner, causing to stop and hold even if an evaluation lane is empty and available has potentially a negative social impact. Therefore, in some embodiments, an alternative artificial risk visualization is preferred in contrast to stop the vehicle which is considered in some instances as to be enough motivation for the driver to avoid excessive reliance on MRM or RMF and abusing system limits of use. In some embodiments, reward and award is at least one of a praise message, credits to use automation or further score to be later provided to the driver as a result of good interaction practice with the automation system, thereby providing some kind of benefit, which serves as incentive for good practice.

[0046] It has further been recognized that it may be possible that regional lawmakers will introduce penalties or fines for drivers who may not properly interact with the automation system as required by regulation. For instance, such an improper interaction may include not responding to a notification to intervene while being engaged in the automated driving, which results in an unnecessary activation of MRM / RFM, which, in turn, causes risks to other road users. However, it may be challenging to confirm that the driver did not try to intervene while vehicle is in operation at high velocity because it cannot be either visually accessible to what is / was occurring in the concealed vehicle cabin nor verifiable of what interaction could have progressed a while ago before the vehicle came to a possible full stop. Thus, it has been recognized that this may be achieved in some embodiments by tracking the movement of the foot. It has been further recognized that the foot movement can be a good indicator with high confidence level for confirming that the driver actually intends to intervene. Additionally, the tracking of the foot movement provides a time-stamped history which can be well preserved and, thus, can provide a reliable means for proving the actions of the driver and the intention or decision to intervene (or to not intervene). Because of the fact that dynamically moving the foot typically requires higher energy than, e.g., rotating the light weighed eye, a stable and smooth motion to lift the heavy foot and bringing it onto active position may only occur once the driver has made an intentional initiative to allocate effort to reach the pedal. Thus, the tracking of the foot movement can provide proof of proper interaction. Compared to the externally observable eyeball movement which requires little intentional effort to make an instant jump for searching visual cues or the movement of the head that may require some additional effort to rotate it horizontally over the neck pivot point, the foot movement detection of the driver provides a higher confidence level of assessing the driver intention to interact. Therefore, in some embodiments, the method, circuitry, device, apparatus and system provide a higher reliability for awaiting the issuing of warning, wherein the appropriate timing of the issuing of warning provides a conditional learning for the as is explained herein and in more details further below.

[0047] Moving the foot from home rest position may require additional intended effort compared to the regular visual search practice exhibited by eye to access visual information, or head movement to reach beyond the search area reachable by just turning the eye. The head movement most likely occurs as an extension of visual pre-verification purpose. For example, looking at the devices for indirect vision (= side mirrors) to see if a vehicle is coming from behind on a nearby lane occurs, but not necessarily because of the clear intention to intervene. Unlike the other behavior, the effort needed for foot transition from home position to acting position requires intentional decision, which allows us to determine whether the driver is properly reacting and making the right decision to intervene in a timely manner, which could be classified as consistent with the design use case. If negligence of the driver persists, in not reacting timely to system notification for the transition to manual driving mode, in some embodiments, the system then triggers the warning and alert to initiate the MRM or RMF with limited chances of the warning and alert being taken as false alarm on appropriate reaction. Keeping appropriate records of this behavior in accordance to each of eventual system notification and warning, allows that these records provide a means to check for a possible violation that the driver may have performed. This allows to verify if each of recorded types of transition event occurred properly or had resulted in violation, e.g., since MRM or RMF has been automatically activated by the system to prevent reaching critical situation in itself if not only classified as poor quality transition. Using this timestamped driver self-behavior information combined with transition quality records, allows to provide transparency to the driver by providing the experienced transition quality with historical tracks, which may serve as an additional visual representation and indicator of the risk to which the driver is exposed. In some embodiments, this is achieved by making the same data simultaneously accessible and retrievable on occasions that the driver gets stopped, e.g., at traffic check points or ordered to stop by officer on routine patrol and requested to confirm appropriate interaction by the driver. In some instances, it is assumed that future vehicles are equipped with an appropriate data storage systems, such that a violation record could be made available either to check for the appropriate interaction between driver and system records or to check what caused the vehicle to have possibly violated base traffic rules to prioritize safety. Moreover it can be checked whether this was caused by an irregular vehicle behavior, and who is responsible for it, e.g. the driver or the vehicle system. Some of this stored information could help the drivers to motivate themselves if it also shows them the risk they face by violating the system instruction to take over control repeatedly, e.g. by displaying the possible penalty of getting fined according to local laws that limit the use of automated vehicles. Showing on the HMI (human machine interface) the consequences of behavior records that lead to penalties for breaking the rules, including some of the previous record, may change how the driver perceives the risk. Just knowing the law that improper use of automation can cause penalties is not enough to motivate the driver's mind directly, but when the violation is visible through recorded evidence, the driver gets a clear sense of actually being fined and this strongly motivates the driver to pay more attention to the situation, and to use their memory to switch from the secondary task to the main driving task more easily. If the driver does not get involved properly, they may face penalties for the repeated minor violations or even lose their driving permission right away depending on the severity of the violation, which is very effective to motivate the driver as an artificial risk without harming other road users.

[0048] Laws defining penalties are not intuitive enough to directly affect driver decision and motivate drive to be respectful of the system instruction. But with the above-mentioned visualization of transition sequence records combined with appropriate interaction violation records, these visual information may provide convincing, intuitive, and effective sense of induced risk perception. This may be essential in at least one of: re-boosting the driver sensory gain, putting efforts in maintaining minimum situation awareness not to get lost in emergency situation, perform appropriate reallocation of working memory in timely manner not to get overloaded before consuming time budget left for transition, and the like.

[0049] In some embodiments, the level of protection for this data from being overwritten until a certain future time or number of events where these stored and older data are overwritten can be parameterized and set based on pre-defined minimum data retention period or number of transition events record according to the benefit of actually encouraging most drivers to be respectful to have a proper interaction with automation system. Data may be temporarily retained for the purpose of traffic control, if allowed by local legislation as a way to prevent repeated violation causing social crises due to increased number of casualties with driver relying too much on MRM or RMF when using vehicle automated driving functions.

[0050] From a human psychological perspective, driver appropriate interaction depends on the driver's ability to balance the goal of achieving a task, such as reaching a destination in time, with the risk involved in driving. The driver thus stays careful enough, but the automation equipped with multiple layers of safety fallback reduces most of the natural sense of risk in manual driving, as explained above. Therefore, some embodiments, to have driver motivation, introduce an artificial but effective virtual risk that challenges the driver to balance it. However, in some embodiments, the artificial virtual risk is just one element to motivate and boost the resultant driver situation awareness and a further element is the automation driving system feedback to the driver notifying, and re-indicating the needs to resume the driving task according to the dynamically changing situation along the drive. The automation driving system feedback may include an initial pre-notification up to warning or alert if late in responding, according to the situation to cope, wherein the warning and alert timing could adversely affect driver behavior development, as discussed herein, which may cause that the driver ends-up untrusting due to repeated incorrectly giving warning and alert timing (e.g., too early or too late), e.g., because the system issued notifications to initiate the intervention independent of the driver intention to cooperate with the system.

[0051] As above mentioned, in some embodiments, the timing to issue a warning and further alert or initiation of MRM / RMF function is elaborated within a shallow or small window of issuing time, since otherwise the driver may not develop an appropriate early initiation of the intervention before reaching the limit of system having to trigger the initiation of MRM or RMF.

[0052] In some embodiments, the reason for the resulting driver behavior is classified, e.g., as bad quality transition behavior in repeated manner, or as an occasional incident resulting in the usage of MRM e.g., as the driver got suddenly ill preventing the driver to perform the intervention requested by the system. If such records of the driver behavior and, e.g., the associated classification(s), are made available and, e.g. easily to retrieve, at the site of traffic control, these system and driver interaction records can serve as proofs of being compliant or a frequent violator exaggerating the use of MRM or RMF in the name of self-safety. The latter case of misuse of the MRM or RMF may consequentially risk other road users, because a vehicle stopped on motorway may cause a high risk of rear end collisions by possibly distracted drivers taken in surprise. Because negligent drivers cause iniquity for other road users, they should be penalized for this type of violation. By seeing the penalty risk, they may be more careful and pay attention to avoid missing important information, improve their memory to stay aware of the situation, and use their working memory better even when distracted by mind wandering. These penalties should not be just abstract information or negligible factors that can be easily ignored. They should be enforced by an effective practice that the law enforcers can implement, if they have the tools and the will to do so, as a way to prevent widespread violations in the public domain. There is a gap between what the regulation forbids when drivers use vehicle automation and their responsibility to interact with it, and how to verify possible violations that may happen at high speed, far from the usual traffic control tools, and concealed within the private cabin of the vehicle. The present disclosure provides in some embodiments a way to check violations easily and to determine their frequency and intentionality, while respecting privacy laws, but also allowing more detailed access if the driver chooses to do so, and presenting the extracted information in a way that makes the driver aware of the risk of being fined, unless they interact properly. The scheme does not allow the driver to blame the system for making incorrect decisions, giving inadequate or wrong notifications, being too early or too late, or anything similar. The recorded data that is easy to access shows what may have happened a long time ago, or the repeated violation, not only to the user of the vehicle but to anyone who can check for violations, which would otherwise be difficult or impossible to verify. The visualization using time stamped interaction data enables checking if the driver's transition behavior was appropriate, as well as rewarding the proactive takeovers and penalizing the lazy or failed transitions due to over-reliance on the automation system, or poor or absent preparation for transitions. When the risk(s) are visualized, the driver can intuitively take preventive actions before encountering the risk. This risk prevention human behavior happens when there is an imminent risk that can be perceived, but such risk may be easily ignored if it requires cognitive thinking to understand the situation.

[0053] Also, as will be discussed below, such data indicating the driver behavior, including data containing the driver decision of the estimated intention to intervene, may be encrypted and privacy may be secured accordingly. Thereby, in some embodiments, only relevant parts of the data may be usable for authorities to check whether an occurrence of driver negligence or delay in properly and timely reacting to notification occurred when the system issued a notification to the driver to regain control of the vehicle. Alternatively, stored data parts with information about interaction violations that do not have privacy sensitive data could be required for law enforcement and quickly accessed by wireless handheld devices. This could allow to immediately display the record of these potential violations that might have happened just before the vehicle stopped. These data records and means for easy retrieval provide transparency to the driver interaction that proceeded between the system and the driver responsible according to system capability. The transparency may provide visualized sense of risk to the driver because the visualized information may provide at least one of: information on the driver involvement in any second task, the system advanced knowledge of the expected time for the driver to resume manual driving according to passive status monitoring, when system made first and successive notification, the first and successive driver response to the system notification, the system successive intervention in the driver negligence to the notification (may such have occurred), preparatory activities the driver took to safely resume manual driving and delay caused by not promptly discarding unsafe activities occupying hands availability to take control. This transparency to the situation, which occurred in the past, provides that all drivers are subject to being checked, and that violation fact occurrence can be promptly checked once a suspect vehicle is stopped for a possible violation. By using wireless communication technologies, the encrypted data may be instantly made available for post stop check if traffic officer wants to check for driver interaction with the automated system, wherein the driver is exposed to get fined if not interacting properly regardless of being directly being observed and witnessed or not. Unlike EDR (Event Data Recorder) data than can only be retrieved on technical service points, in some embodiments, there is no need to physically connect any cable enabled a practical and timely confirmation of any irregular or illegal used of the vehicle system. Data, which is locally encrypted, provides driver privacy but can still give transparency to any occurrence of inappropriate interaction that may result in automation use violation or abuse like continuing a second task. This scheme prohibits the driver to excuse and deceive penalties. Running a violation prevention campaign by checking the records reveals the driver behavior neglecting the system request of intervention.

[0054] The records and the included data allow to extract the interaction violation, the associated point in time, associated system capability and the associated point of time of notification / warning , which each or together enable the verification of violation. Other raw or pre-processed data stored in encrypted manner prevents any malicious use of such data. In some embodiments, such (encrypted) data is still accessible in a situation of need, such as in the case to prove driver innocence, or in the case that is to be determined whether the system violation is to be registered. This is very effective and creates a virtual risk for the driver that any violation will have consequences, even if an officer on patrol does not see it live. Data that is recorded and displayed locally can help the driver avoid accidents by showing them the risk level, while also protecting the privacy of the vehicle's internal image from being transferred or stolen by someone nearby who might have a harmful motive. Typically, no one wants their post-accident condition inside the cabin to be widely shared for reasons other than rescue.

[0055] Therefore, some embodiments pertain to passenger safety circuitry for warning an operator of a vehicle with an autonomous driving function, the circuitry being configured to: determine an operator readiness level based on an operator foot position; determine whether an operator warning is to be issued; and based on the determined operator readiness level, issue the operator warning.

[0056] In some embodiments, the determination whether an operator warning is to be issued includes also the determination whether the issuing of the warning is to be suppressed. The determined operator readiness level may include also a delayed operator reaction.

[0057] Circuitry may pertain to any entity or multitude of entities which digest data and output a warning as discussed herein. The circuitry may include a processor, such as a CPU (central processing unit), GPU (graphics processing unit), or a combined CPU / GPU, as well as an FPGA (field-programmable gate array), or the like. In some embodiments, the circuitry further includes image acquisition and / or depth estimation means in order to determine the foot position including or additionally its initial muscular activities as indicator of the driver actual intention to intervene. In some embodiments, the circuitry further includes means for outputting a signal, such as a loudspeaker, an optical notifier, a vibration notifier, a haptic notifier, or the like.

[0058] However, the present disclosure is not limited to the case that the circuitry also includes the input and output but since it may be sufficient that the circuitry according to the present disclosure includes data processing means for determining the readiness level, determine whether the warning is to be issued, and issue the warning based on the readiness level, as will be explained below. Hence, according to the present disclosure it may be necessary that foot position is input, but it may not be necessary that the circuitry according to the present disclosure also acquires the foot position. For example, the circuitry may be configured to be interoperable with circuitry that acquires the operator foot position.

[0059] The circuitry may be configured for passenger safety, i.e., may include a passenger safety circuitry, and may thus be used in that context. The circuitry may be configured issue a safety function for a passenger (e.g., a driver or any other passenger) of a vehicle (e.g., a car, a bus, a boat, or the like), wherein a safety function may be embodied in various ways, such as a warning to the driver (e.g., an operator warning) in form of a sound, a vibration, an optical signal, a haptic signal, or the like. The safety function may further include an automatic braking of the vehicle, an automatic bypassing of an obstacle (for example by steering control), or the like.

[0060] The vehicle may have an autonomous driving function, i.e., may be configured to drive at least partially autonomously, and the driver may have the possibility or may be obligated (e.g., depending on a jurisdiction) to intervene while the vehicle is driving autonomously.

[0061] In some embodiments, the circuitry may be configured to determine an operator readiness level based on an operator foot position. In such embodiments, the foot position of the operator may indicate whether the operator is willing to intervene. It has been recognized that a displacement of the foot (or a lifting of the foot) may be a more reliable indicator of operator readiness than, for example, the operator’s line of sight. Moreover, it has been recognized that it may be easier to detect the operator’s foot, than for example, the operator’s eyes. Furthermore, it has been recognized that it may be easier and accurate to detect the operator's foot than, for example, the eye gaze.

[0062] However, the present disclosure is not limited to the case of detecting the operator’s foot position or displacement since any body part or body feature which indicates a high confidence for the operator’s readiness may fall under the scope of the present disclosure. For example, eye, face, body posture, hand, legs, or the like, may be used as an indicator. For example, the legs or certain muscles of a leg may be recognized in order to determine whether the driver intends to intervene. Other biological data sensing technologies are also known. Such other biological data sensing technologies may be considered as being invasive, which may result in a lower acceptance of the user in general. In some embodiments, such biological data sensing technologies provide a solution for detecting early cue of driver decision to initiate a requested interaction. Also, the hands and / or forearms may be used as an indicator of whether the operator intends to use the steering wheel (for example when the hands / forearms move into the direction of the steering wheel). Also, determining whether the seat belt is used (or correctly used), whether the second task is aborted, the operator’s breath / heart beat (or the like), whether the operator is in driving posture, whether the eyes gaze on the road, whether the hands are on the wheel, whether the operator looks at mirrors to check traffic, the driver’s response to a notification / warning may be used as an indicator. Moreover, it should be noted that the detection of a body feature may depend on user preferences such that, instead of detecting the foot, the hand (or the like) may be used as an indicator.

[0063] Furthermore, different drivers with different physical or mental capabilities may have disabilities as well as preferences. Thus, in some embodiments, automation function is used as complementary function to accomplish mobility where its primary means of interaction may differ from most commonly fond drive behavior and interaction where the right foot is commonly allocated to activate the brake and / or acceleration pedal, or hand to control the steering wheel. Therefore, the primary source of driver intention could be weighted to other source in such case, in preset manner, or further by system learning function along repeated use to best perform estimation accordingly.

[0064] Any indicator may be used in a time stamped manner to determine whether the driver has the intention or is ready to intervene if necessary.

[0065] An intention to intervene may refer to a mental state of the operator and it may not be possible to directly determine such an intention, since it may only be determined if not estimated indirectly based on indicators such as the foot position and motional nuance or based on any other reaction of the operator. Since the present disclosure gives the possibility to detect the intention to intervene, there may be a rather low false positive rate (thereby preventing the system to become a “cry wolf system” whose warnings the operator might ignore at some point - or in extreme cases, the operator may develop symptom similar to an “oppositional defiant disorder”) against the system warnings since when it has been recognized that the operator intends or is ready to intervene, a planned warning may be suppressed or it may be determined that a possible future warning may not be necessary, according to personal behavior learning results.

[0066] Furthermore, in some embodiments, additional fail proof data analyses are implemented as part of the warning and alert to suppress function and avoid misjudging driver unintentional leg / foot lifting which may occur just after waking-up, leg spasm, or as a result of mental disease causing occasional spasm or a like, which otherwise could be misinterpret by the system as driver intention to intervene and accidentally suppress issuing a warning when desired to. Several procedures are adopted in some embodiments to judge whether it is a continuation of the driver response to notification to intervene, wherein this may be implemented based on a procedure of analytical consideration, which is initially started by detecting driver reactive response to the notification. This includes eye gaze tracking exhibiting a visual confirmation for the road scene ahead, gaze to the side mirror, if applicable, or the like, because, in some instances, the driver will not just directly make a decision to actuate a pedal without gazing or start interpreting the situation to be coped gazing around.

[0067] In some embodiments, based on the operator foot position (or any other indicator mentioned herein), the operator readiness level may be determined. For example, an amplitude, a velocity, degree or the like, of a foot displacement may indicate the operator readiness level. For example, the higher and / or the faster the operator lifts the foot, the higher the readiness level may be determined without limiting the present disclosure in that regard. Readiness level may depend on a whether toe part, heal part or entire foot is lifted.

[0068] In some embodiments, it may be determined whether an operator warning is to be issued. The determination may depend on factors external to the vehicle, such as an amount of traffic, roadside transmitters issuing information to the vehicle and / or on factors internal to the vehicle, such as a state of the operator / driver (e.g., the operator may be sleepy).

[0069] In some embodiments, the operator warning is issued based on the determined operator readiness level. For example, if it has been determined, based on the foot position, that the operator is willing to intervene (e.g., hit or apply the brakes), the operator warning may not be issued. On the other hand, if it has been determined that the operator readiness level is too low, the warning may be issued.

[0070] In some embodiments, the operator readiness level is determined based on the operator foot position at two different points of time. For example, based on the determination at two different points of time, a velocity and / or a displacement may be determined.

[0071] Hence, in some embodiments, the circuitry is further configured to: detect a foot displacement between the two different points of time, wherein the operator readiness level is determined based on the detected foot displacement.

[0072] In some embodiments, the initial and proceeding motion of the driver foot is detected using an event trigger camera system. The event trigger camera system may be configured to (immediately) detect the temporal changes resulting in the initiation of the foot transfer from home rest position to active operation point. In some embodiments, this enables early detection of the driver intention to intervene. In some embodiments, the operator foot position is determined based on a reference point in the vehicle.

[0073] For example, the reference point includes at least one of a pedal, a home position optionally including a foot rest, a user defined position including at least one of a comfortable position and a normal position, a history-based position, and a position determined based on an artificial intelligence.

[0074] In some embodiments, the circuitry is further configured to: issue the operator warning, if the readiness level is below a predetermined threshold, as discussed herein.

[0075] In some embodiments, a type of the operator warning is based on the determined readiness level. The type may include a sound signal, a light signal, a vibration, or the like, or a combination thereof as discussed herein. If it is determined to issue a warning, different types may be associated with different readiness levels (e.g., ranges of readiness level). The type and / or intensity of the warning may further depend on a situation. For example, if a traffic situation is determined to be very dangerous, a very loud warning may be issued, if the readiness level is rather low, a relatively less perceptible warning is issued. If the readiness level is high (but not high enough that no warning is issued), the sound signal may not be as loud. On the other hand, if the traffic situation is determined to be not dangerous (but still sufficient to issue a warning) and the readiness level is rather high (but not high enough that no warning is issued), a soft vibration may be issued. Several combinations and types of warnings may be apparent to the skilled person which shall fall under the scope of the present disclosure without the need for explicit mentioning of such combinations. In some embodiments, the circuitry is further configured to: update a warning database based on whether a warning was issued. Some drivers may try to keep continuing a second task and intentionally deceive the system by pretending to lift the foot and suppress the issuing of the warning. Deceiving the system detection function by pretending to move his foot to active position or generating initial motional nuance but not actually reaching to actuate a brake or accel pedal shall be prevented in some embodiments. However, these practices are not only undesired and may lead to reduced time budget in worst case and, thus, may also result in system delayed triggering of the deployment of MRM or RMF, which in turn may result in unnecessary hard deceleration or the like, if this does not even causes a critical situation in worst case. Therefore, the system may perform an individual behavior analyses enabling to differentiate and discriminate from deceiving prominent motion behavior and the mentally intended action by taking into account the resultant successful and deceived behavior, e.g. by adding issuing a penalties score adopted on the action classified as deceiving or not from each accomplished transition required event. An important targeted aspect of this discrimination may not be a perfect discrimination of the deceiving procedure but rather the avoiding that the driver develops a repeated habitude to deceive. While trying to deceive, the driver is conscious of the situation required to cope and the driver will likely not to cause a critical situation.

[0076] For example, such data may be used for future situations in which a warning needs to be issued. For example, a type of warning may be changed in a similar situation, because the operator did not react to the warning. On the other hand, it may be recognized that the driver has reacted before the warning was issued such that in a similar situation, a warning may be delayed or suppressed as it is most likely that driver will properly react. This kind of enhanced learning could improve the recognition of the driver’s real intention to intervene. Moreover, from the driver’s psychological perspective, it functions as operant conditioning that rewards correct positive behavior and punishes deceptive behavior that aims to postpone the warning time, such that the warning may be delayed or suppressed.

[0077] In some embodiments, the circuitry is further configured to: update a motion database for determining a future operator readiness level.

[0078] For example, determining an operator's typical movements may be used to discriminate the typical movements from an actual intention to intervene. Hence, the determination of the operator readiness level may be adapted and individualized based on, e.g., detailed behavior characteristic which may be learned on such historical data. This personal characterization and dynamic learning may be important in some embodiments, because the physical and mental capabilities change over time and differs, e.g., from young to elder, and the mental capability may have further individual tendencies of development. Therefore, in some embodiments, a uniform threshold used to judge, such that an intention to intervene is uniformly detected. However, in such embodiments the estimation of the driver readiness could not be achieved with precision in some instances and be prone to repeated error and inappropriate alarm.

[0079] In some embodiments, the operator readiness level is further based on an emotional state of the operator.

[0080] For example, recognition of the emotional state may be used to refine the determination of the operator readiness level. For example, if the foot is lifted, but it is recognized that the operator is distracted and the lifting of the foot is because the operator moves it with music that is playing, the readiness level may be falsely determined as high. Hence, the recognition of the emotional state may correct this determination and thus, the circuitry may re-determine the operator readiness level as low.

[0081] For example, considering the personal effect of mental stress, workload, anxiety, immersive situation into a second task which may include watching a live sport match, engaged in gaming, a tele drama and any other emotional condition may cause different effect besides the intrapersonal behavior characteristic difference. Therefore, it has been recognized that an emotional state may be taken into consideration and that, for example, an Al learns a personal / individual reactive difference to a notification which occur for different incidences requiring transition of control from system to driver. According to the present disclosure, any Al may be used which is capable of learning and generating data needed for the data discussed herein. However, it may be important that a proper traceability of captured information is maintained for post evaluation, wherein still a certain level of user privacy is maintained, wherein also efficiency to achieve low carbon emission may be taken into consideration. Also, traceability of the Al decision taken may help to gate or block undesired learning where an added assessment flag is provided in parallel to the Al internal state to enable monitoring of the learning process. This assessment flag may enable for a detection of unreasonable decision taking, but still satisfying the target outcome. The gating capability to enable or disable certain Al generated decision taking may be important in view of ethical and moral schemes and / or laws.

[0082] Some embodiments pertain to a passenger safety method for warning an operator of a vehicle with an autonomous driving function, the method including: determining an operator readiness level based on an operator foot position; determining whether an operator warning is to be issued; and based on the determined operator readiness level, issuing the operator warning, as discussed herein.

[0083] In some embodiments, the operator readiness level is determined based on the operator foot position at two different points of time, as discussed herein. In some embodiments, the method further includes: detecting a foot displacement between the two different points of time, wherein the operator readiness level is determined based on the detected foot displacement, as discussed herein. In some embodiments, the operator foot position is determined based on a reference point in the vehicle, as discussed herein. In some embodiments, the reference point includes at least one of a pedal, a home position optionally including a foot rest, a user defined position including at least one of a comfortable position and a normal active position suited for a manual driver, a history-based position, and a position determined based on an artificial intelligence, as discussed herein. In some embodiments, the method further includes: issuing the operator warning, if the readiness level is below a predetermined threshold, as discussed herein. In some embodiments, a type of the operator warning is based on the determined readiness level, as discussed herein. In some embodiments, the method further includes: updating a warning database based on whether a warning was issued, as discussed herein. In some embodiments, the method further includes: updating a motion database for determining a future operator readiness level, as discussed herein. In some embodiments, the operator readiness level is further based on or biased by an emotional state of the operator, as discussed herein.

[0084] The methods as described herein are also implemented in some embodiments as a computer program causing a computer and / or a processor to perform the method, when being carried out on the computer and / or processor. In some embodiments, also a non-transitory computer- readable recording medium is provided that stores therein a computer program product, which, when executed by a processor, such as the processor described above, causes the methods described herein to be performed.

[0085] The present disclosure may generally be applied in the following exemplary use-cases:

[0086] • Detection of driver (operator) tending not to be ready for possible automatic intervention: o Vehicle issues notification of need that the driver should take control, timing based on periodically detected driver steady state and localization o In response, detect the driver response and further in footwell feet movements indicative of readiness to take control o If not detected (or driver detected to be non-responsive), apply autonomous safety measure (e.g., stop / evade)

[0087] • Timestamp to detect and enable judging that driver is in control in timely manner: o Detect driver's foot in home position (e.g. on foot rest, by computer vision or pressure / detection sensor) o Vehicle issues notification of need that the driver should take control o Judge foot movement and determine that driver is exhibiting an intention and / or in a state ready to take control o Record timestamp of the point of time that it has been determined that the driver is ready to take control, or when it exhibited initial signal of the intention to intervene

[0088] • Driver reacts before circuitry (e.g., Al) reacts - Determine that the driver is in control: o Detect foot movement from home position and generate a first timestamp o Vehicle system provides and advanced pre-notification of the incoming / approaching events or milestone point along the planned trip o Vehicle issues notification of need that the driver should take control o Determine whether detected foot movement puts driver in a ready state o Generate a second timestamp o Information / timestamp(s) is stored in an immutable way, for example using hashed data and deleted a certain time after the incident

[0089] • Differentiate natural foot movement from driver history from foot movement indicating readiness. Foot movement is assumed to be faster if genuinely reacting

[0090] If the present disclosure is implemented based on an artificial intelligence (Al), the following explications may apply:

[0091] The Al system may for example take decisions based on image data or other sensor inputs.

[0092] Al decisions may be taken by different circuits in multiple different parts of an overall larger system and there may be some redundance as to which decisions have contributed to an output action. Since trained Al models may vary over time it may be impossible to recreate or simulate a decision taken at a later time even when having access to ground truth data. Al Decisions may be used to reinforce training data for the Al model and may be fed back into the Al model.

[0093] Logging of input data and output data and possibly intermediate data as well as state data for the Al model may be used. However, logging of such data may create large amounts of data - possibly too large or costly to store.

[0094] In one example log data may only be stored when a decision is made which is close to (within a threshold) of being decided one way or another. Tri-state or n-state decisions may be equally applicable. Also, ground truth data, such as an image or multiple frames of video may be stored in association with the log data.

[0095] On the other hand, rather than storing the entire ground truth data for a decision, a compressed version may be stored to reduce the amount of stored data since it may be desirable that it is still possible to recreate the decision from the compressed version. For example, according to the present disclosure, the ground truth data may result in (in a descending order with respect to possible data size, i.e., as a hierarchy):

[0096] • Storing of an uncompressed video

[0097] • Storing of a compressed video

[0098] • Storing of a compressed video that is reduced in dimension

[0099] • Storing of only video luminance data (e.g., black and white)

[0100] • Storing of only segmented objects from the video

[0101] • Storing of only feature data such as color statistics, or downsampled feature data

[0102] • Storing of only detected body skeleton joint articulation point data

[0103] In some embodiments, the log data may be reduced in size by storing the ground truth data at a level in the hierarchy proportional to a confidence in the Al decision that was made, for example in proportion with a binary Al decision.

[0104] Some embodiments pertain to vehicle (e.g., car) equipment including: a foot rest for an operator of a vehicle, wherein, when the operator places the foot on the foot rest, it indicates that the operator intends to rely on an automated driving function; and a foot tracker configured to detect the operator’s foot and to determine whether the operator places the foot on the foot rest. The foot tracker may be further configured to determine whether the operator intends to use a pedal, or other actuator intended to be operated by foot, of the vehicle, for example based on intentional nuances of the operator, as discussed herein.

[0105] The foot tracker may be further configured to detect a signal of a muscle to initiate a lifting of a leg and / or the foot, e.g., based on at least one of a sensor that is attached to the driver, a depth sensor (e.g., time-of-flight or infrared), a two-dimensional vision sensor, a radar signal, a proximity sensor, a pressure sensor, an event triggered type image sensor or the like.

[0106] The intention nuances may be estimated based on an Al that evaluates the very early motional nuance of the operator, such as a motion of at least one of a leg, at least a portion of a leg such as thigh or calf, a toe, a face, an eye, a head, and / or a change in body posture. The Al may learn to estimate the intention nuances based on historical records of the operator in which incidents may be used as ground truth in which the operator intervened in automated driving. In some embodiments, the Al also learns personal characteristics, wherein the behavior may differ from driver to driver. Generally, several different cue may be used as intention estimator. In some embodiments, the foot lifting is used as cue as the foot lifting from its home position is the type of driver behavior that may require a significant energetic resource of the operator compared to, for example, turning the eyeball or turning head to road ahead. Therefore, in some embodiments, the foot lifting action is considered as an indication that the operator had made an actual decision with intent and, thus, may serve a as a reliable indicator with low rate of false detection. Moreover, once the foot is lifted it needs to find a stable landing (acting) position.

[0107] The vehicle equipment may further be connectable or include recording means for extracting or generating time sequential key parameters to be stored, thereby having a capability to recover a time stamp of each action and learned decision taken by the Al, thereby providing the possibility to track how the Al has evolved over time according to an adapted behavior that the operator exhibited along the continuous use of automation.

[0108] Some embodiments pertain to a non-volatile memory for maintaining record of pre-notification timing, notification timing, warning timing, alert timing and how or what triggered the system (e.g., the Al) to emit each of the respective signals in connection to the detected foot position of the operator. The Al may learn and adapt when it is appropriate to issue the pre-notification, notification, alarm, and the like, based on the operator’s behavior and historical records. The Al may learn and adapt differently according to the type of alarm and or specific scenario in which it is triggered or deployed. The adaptation may depend on an operator’s reaction speed, the operator’s sense of urgency, and the like. Thereby, a successful transition from automated to manual driving may be achieved.

[0109] The warning timing window may be rather narrow and may be dynamic and individual to the operator. It should be implemented such that the notification is not issued too early such that the operator learns to ignore the warnings (or gets too comfortable due to long time margins with little risk), and also not too late that the operator cannot follow an instruction. After the warning timing window, an alert window may open which may include a final alert within a predetermined time and after that, an initiation of MRM / RMF before a time critical point is reached. Before the warning timing window, a notification window may exist and before that, a window for a soft pre-notification.

[0110] However, in the warning timing window, the warning should be issued early enough that a safe driving is still possible. It should also consider that the driver may need a certain time to react and to resume to manual driving. The current time window may depend according on its initial status. However, it may be challenging to detect a preparatory stage and predict a clear intention of the operator to resume to driving task which is why according to the present disclosure, the operator’s foot initial motional nuance is detected in some embodiments. Otherwise, if too much of the time-budget is used for giving the operator enough time for preparation, the vehicle may be continuing its trip reaching its limit and requiring to be brought to an abrupt halt because RMF / MRM may be activated as a safety measure.

[0111] In the following, an embodiment is given of how a smooth transition from automated to manual driving may be achieved:

[0112] Continuous information of environment indicative of oncoming situation is obtained in a simplified and concise HMI (human-machine interface) so that driver get overloaded to process

[0113] Operator status is analyzed to estimate the time required to resume to manual driving taking into account the current status of the operator (for example how alert the operator is)

[0114] - Evaluate the required operator’s successful recovery ratio to manual driving when crossing a pre-determined point for notification or reaching point of interest (a point on the travel path determining a certain rate of transition failure allowance, and to successfully achieving manual transition besides still triggering the MRM / RMF although not preferable to be over relied due to its side effect the MRM causes to a fluent surrounding traffic flow). In some embodiments, the use of MRM is enabled, since the MRM may be required because human and machines are not perfect and there exist always a room for failure. In some instance, the point of interest is the location where the vehicle would make an emergency stop, since the driver fails to accomplish a safe transition to manual mode in advance to reaching a safe critical point of the planned path.

[0115] Determine an early notification point (in order to minimize the use of MRM / RMF)

[0116] - Detect response of the operator to the notification for the system to confirm that notification was properly and consciously accepted and perceived, thus not missed.

[0117] - Re-adjustment of strategy according to the detected operator response (including possible deceleration to gain longer time budget, if response indicative of not being accurate and low confidence)

[0118] If warning point of time is reached or indicative of decreased confidence to have properly perceived the notification, further monitoring of foot initial motion is carried out as an indicator of an intention to intervene, as discussed herein, and suppress warning depending on the monitoring results (the intentional motion detected or not), or issue a warning if not detecting the intention as expected within the time window allowance

[0119] Confirm whether foot has reached pedal for activation of pedal brake or accelerator

[0120] - Evaluate operator’s reaction: Evaluate the overall transition and determine whether operator has been pro-active, succeeded or has failed to react properly. This evaluation may be used in order to decide whether the operator is allowed for future use of automated driving, and / or to determine future time budgets individual for each operator, and / or for updating a database, as discussed herein.

[0121] In some embodiments, an emotional evaluation / correlation may be carried out between multiple types of data relating to the operator, thereby reflecting emotional influences that may distract the operator from the driving task or impact their alertness.

[0122] In some embodiments, it may be evaluated whether the driver successfully intervened, thereby achieving positive credit (e.g., as input for the Al to create the individual characterization dictionary to generate successful transition delay distribution profile), if the intervention was successful and appropriate. Otherwise, negative credit may be given or penalty points may be accumulated (e.g., as input for the Al). Also, based on the credit or penalty, an atmosphere may be created in which the driver’s motivation to react is increased, e.g., when positive feedback is given based on positive credit and when negative feedback is given based on negative credit. Credit level may be reflected in a scoring system, for example using points. In some embodiment, the Al may be configured to generate and use a behavior dictionary of the operator to predict an operator behavior. The Al may further use a level of stress, anxiety, drowsiness, attention / inattention, occurrence of reckless reaction (such as a verbal or gestural outburst), and the like.

[0123] Returning to Fig. 1, there is depicted a passenger safety method 1 according to the present disclosure in a block diagram.

[0124] At 2, it is determined whether the foot is at a predefined position, such as a home rest position, as discussed herein. A body feature tracker may, in some embodiments, additionally record initial stages after the driver is recognized.

[0125] At 3, an early intention estimation is carried out which detects nuances of an initial foot motion.

[0126] At 4, it is discriminated or determined whether the estimated nuances could mean that the driver intends to resume to manual driving.

[0127] At 5, a behavior analysis is carried out which uses at least one of an eye position / glance / saccadic movement, a head movement, and the like.

[0128] At 6, a foot position / displacement is determined and it is evaluated whether the operator has lifted the foot as a cue for whether a planned signal is to be suppressed.

[0129] Based on this determination, at 7, it is evaluated whether the operator has an intention to intervene. If it is determined that there is no intention to intervene, but it would be necessary to intervene, at 8, a safety function is activated, such as MRM or RMF. Moreover, a violation record is updated. If yes, at 9, the transition is evaluated and it is determined whether the transition is smooth or abrupt (in some embodiments, a quality of transition is determined). Based on this, a credit record is updated.

[0130] At 10, based on the updated records, an Al dictionary is updated, as discussed herein. The dictionary is updated, such that the best notification timing based on successful or failed transition is re-established, further taking into consideration detected stress / emotion, drowsiness level, and the like. Hence, at 11, an emotional evaluation is carried out for updating the dictionary. The Al dictionary enables to estimate an individual time budget depending on the emotional status and depending on the initial foot motion and possibly other supportive cues.

[0131] Moreover, a driver behavior dictionary may be generated or updated which may be fed into the early intention estimation 3. Moreover, a time stamped decision record may be generated or updated, as discussed herein. Fig. 2 depicts a further passenger safety method 20 according to the present disclosure in a block diagram.

[0132] At 21, it is determined whether the operator’s right foot is at a predefined rest-position. If the operator places the right foot on the rest position, it is indicated that the operator rely on automated driving.

[0133] At 22, a visual and / or vital detection of muscular (e.g., quadriceps femoris) activity to lift the leg and / or rotate the foot is carried out and it is determined whether this muscular activity means that the operator intends to take over, at 23.

[0134] Hence, at 24, it is evaluated, whether the operator intends to intervene as an early estimation. If no, the method goes back to 22. If yes, the footwell is 3D-tracked at 25 to determine whether the operator lifts or rotate the foot in order to reach a pedal.

[0135] Based on the determination of 25. At 26, a driver dictionary is updated, as discussed herein, which is fed back to the early intention detection 23.

[0136] Moreover, at 27, time stamped behavior records are generated and encrypted for Al learning validation check. At 28, time stamped records for driver intervention behavior quality check for incentives and penalties are generated, as discussed herein.

[0137] It should be noted that according to the present disclosure, personal data of the operator or any passenger may be protected. For example, non-driving related data may not be collected or at least it may be transferred in an appropriate manner. One aspect pertains to the collection of data indicating whether the operator follows instructions given by the system (or whether the operator fails to do so).

[0138] For example, the following data may be collected:

[0139] - Notification type and timing, including information of severity and urgency of the notification; in some embodiments additionally a past cumulative violation record indicating whether the operator has understood past (similar) notifications

[0140] - Response reaction taken by the operator and status of cancelation of any NDRA (nondriving related activities), if any

[0141] - Record of emission of warning and alerts indicating whether the operator has failed to promptly react to a notification

[0142] Classification of NDRA (e.g., texting, web browsing, playing, shopping, drinking, etc.) Type of violation (e.g., first time violation; if not first time, record of past violation; repeated violation, violation severity etc.)

[0143] Occurrence of past light / moderate violations (how long has it been since the last violation?)

[0144] Occurrence of past severe violation under which it is worth considering disabling access to automatic driving function

[0145] Such data may be written back to database, i.e., the database records may be updated or created with such data or data derived therefrom. Also, a failed transition history may be generated and stored without disclosing the operator’s identity.

[0146] A local storage may be provided with personal records that the operator may have access to, thereby giving the operator a possibility to prove the driving behavior, e.g., before of a court or other body such as insurer (e.g., if required to defend, but the records are disclosed under the full control of the operator).

[0147] Fig. 3 depicts a block diagram of a method 30 for determining an additional risk factor based on an evaluation of an operator’s emotional status. It has been recognized that the operator’s emotional status may affect a decision (or a speed of the decision) that the operator makes, thereby affecting a time budget that is used for the waming / alert / notification. For example, an operator with high anxiety, stress, and the like, may not spend as much thought on how to properly react to a situation and might miss a warning signal.

[0148] The additional risk factor may be used in order to adjust a notification / warning / alert. For example, a timing, a type, an intensity, or the like may be adjusted. Moreover, based on the evaluation and / or the risk factor, a safety lag gap time margin may be adjusted. For example, a predefined time budget bias may be adjusted based on the risk factor before the system reaches a critical point. The operator’s emotional status is used in some embodiments to perform the notification adjustments, and, additionally, its result may be used to actuate preventive advanced relaxing function, which is generally known to the skilled person. Thereby, the risk may be removed or mitigated which otherwise should depend only on the time budget adjustment.

[0149] At 31, an input of a wearable device, such as a smartwatch, is acquired. The data of the wearable device is indicative of data representing at least one of sleep level (e.g., sleep deprivation), a heart rate, an anxiety level, body temperature, a secondary task, and other risk data to be considered when using automation.

[0150] At 32, the data of the wearable device is correlated to the operator’s emotional status. At 33, data of an Al verbal chat is acquired. This data represents a speech chat between an Al virtual assistant and the operator. Based on that data, it is assessed how the operator reacts verbally which through its auditorial and verbal analyses may indicate an emotional status of the operator.

[0151] At 34, a physiological motoric state of the operator is obtained. The motoric state may indicate the operator’s ability to use, for example at least one of a steering wheel, a pedal, or the like, as an indicator of the emotional status.

[0152] In some embodiments, depending on availability of prolonged use of automation in a specific situation, the driver may even exhibit numbness when crossing legs or arms for long periods of time. This may require an intelligent management to periodically assess and prevent the deterioration of motoric state or keep the deterioration within controllability allowance. This can be made available when vehicle is determined to be within an MRM activation allowed segment with no major risk if vehicle is forced to come to a standby due to driver failure to properly take over control.

[0153] At 35, the data obtained in 31 to 34 is used for an evaluation of the emotional status based on an Al that is trained to do such an assessment. For the simplicity of the explanation, each of incoming components from 31 to 34 is shown as independent input components but the emotional estimation performance is better achieved in some embodiments by further analyzing the correlation observed in between these components, e.g., when driver is angry and stress may exhibit a slight delay in reaction time, sometimes resulting in over maneuvering to compensate for initial erroneous maneuver as a result for mental resource deallocation to task other than the driver's regularly dedicated driving task.

[0154] At 36, based on the emotional status, the additional risk factor is determined based on an Al that is trained to do such an assessment. The emotional estimation is not necessarily available in all situations. The availability of devices and accessibility to possibly collect data elsewhere may depend on individual preferences to wear and take life log using wearable devices or other vital data sensing devices. Therefore, it has been recognized that it may be difficult to adopt a unified method and measure as well as a threshold on classifying to determine the emotional level of every driver in a unified manner. However, if the accessibility of such vital data is made available by the driver to the vehicle operation system, in some embodiments the automation management device may use an Al agent for the individual characterization process, to generate a personal dictionary to enable estimating the driving task performance indicative of the bias required to compensate the driver deviated behavior caused due to the present emotional status. Fig. 4 depicts an illustrational diagram 40 representing how a positive loop may be created for the operator. According to this embodiment, the transition from automated to manual driving is considered as a cyclic procedure. It is assumed that road environment is heterogenous and a variety of different situations may occur based on which the operator may need to resume to manual driving.

[0155] The diagram 40 includes several rings 41 to 44 which represent different escalation level from safe (the innermost ring 41) to dangerous (the outermost ring 44). Each ring 41 to 44 is given a predetermined time budget which may depend on the specific situation. Also, the time budget may be different for each of the rings 41 to 44. The time budget is represented with the lines 45 and 46, wherein a decision point after which a notification / warning is issued is represented with the line 47. The line 47 represents the point of time when a certain cycle has reached its time budget and it is then decided, whether and which notification / alarm is issued.

[0156] During the time budget represented by the ring 42, a pre-notification is issued in the beginning and it is assessed at the time represented by line 45, whether a smooth transition has happened or is about to happen. If yes, the operator is rewarded, as discussed herein. If not, in the time budget represented by the circle 43, at first, a soft warning is issued before the line 46 and when the line 45 is reached, a hard warning is issued. If the driver still does not respond, during the time budget represented by the circle 44, an alert is issued that MRM / RMF is activated, but the operator has still the possibility to react and transition into manual driving.

[0157] During the respective time budgets, an evaluation on the transition quality and the driver response is carried out, as discussed herein. As also discussed herein, respective notification / warnings / alerts are suppressed if it is detected that the operator lifts the foot in order to resume to manual driving.

[0158] By the cyclic iteration of notifications / warnings / alerts, a pro-active behavior of the operator may be enhanced since the operator may learn that an annoying warning may be suppressed due to their behavior.

[0159] Fig. 5 depicts a timeline according to the present disclosure showing which action is taken at different points of time before a critical point (overall time budget) is reached before safe continuation of the trip cannot be guaranteed anymore. As discussed above, an escalation of notifications may be carried out.

[0160] The first level includes the issuance of a soft attention earcon (earcon: like an icon but auditory, i.e., for the ears) indicating advanced information of the road ahead, thereby creating situational awareness for the operator, such that the operator is cautious. Jingles may also be used and recognized by the operator.

[0161] At the second level, a pre-notification is issued in the form of a soft attention earcon (possibly another sound than before) which helps the operator to initiate for an upcoming transition.

[0162] At the third level (not shown), a notification is issued, again in the form of a soft attention earcon (possibly another sound than before).

[0163] At the fourth level, a hard attention earcon is issued as a pre-warning which is then further escalated to a warning including visual and acoustic signals which the operator perceives as more and more annoying.

[0164] If the operator does not react, the time window for safe MRM starts, i.e., MRM which does not cause risk to other road users, and after that, the window for risky MRM starts before reaching the critical point.

[0165] Fig. 6 depicts an interior of a vehicle 60. An operator 61 is monitored with a plurality of cameras 62 to 64, one of which is (camera 64) directed towards the footwell of the vehicle 60, thereby detecting the operator’s legs and feet. Another camera 62 monitors the driver from the front and the other camera 63 monitors the driver from above. The cameras 62 to 64 may be part of circuitry to determine the operator’s body features, as will be discussed below. Thereby, it is possible to determine 3D position / movements of the lower legs by the camera 64, to determine hands, arms, and upper leg movement by the camera 62, and to determine head movements by the camera 63. The cameras may be configured to determine a depth image and a reflection image to determine the respective movements, as will be discussed below.

[0166] Fig. 7 depicts a flow chart of a vehicle safety method 70 according to the present disclosure.

[0167] At 71, an initial driver state and Al dictionary with past records and an emotional log are obtained. The driver state is obtained with at least one of a wearable device, a hand-held device, an emotional analysis (e.g., via in-cabin facial analysis, voice analysis, or the like), medical records, and personal takeover behavior. The past record data is obtained from a database, as discussed herein, which is updated regularly (e.g., based on issued notifications and / or driver behavior) and / or from any of the sources which are used for obtaining the driver state. The emotional analysis may be carried out, for example, by a voice analysis, face analysis, electrocardiogram, breath analyzer, body temperature, based on medical records, or the like. For example, in 71, it may be determined that the driver is a relatively old driver with a slow response, who is distracted and sleep deprived and has a high workload. In some embodiments, the driver state is determined based on at least one of the following: monitoring of emotional status, monitoring of drowsiness level, monitoring level of NDRA involvement, vital sensor monitoring, monitoring of face, monitoring of head, monitoring of (visual) attention, monitoring of driver’s response over handheld application, monitoring of hands, monitoring of steering wheel and respective torque detection. The emotional status and / or the drowsiness level may further be used for recalibrating the system (and / or the time budget). For monitoring the driver’s response over an application, different applications may run in parallel to confirm the driver’s response, which requires mental awareness to make a decision to correctly answer back.

[0168] At 72, a negotiation on the target time budget is carried out before MRM / RMF is issued. The negotiation depends on a risk that is associated with a present or potential situation which may be determined, for example, based on road conditions, or the like. For example, in a tunnel, MRM may need to be avoided. The negotiation is carried out based on data obtained from at least one of a navigation system, a driving automation management system, an LDM (Local Dynamic Map) (e.g., including information about the road, such as presence of MRM / RMF evacuation lane, evacuating spot (such those encountered withing a long tunnel) availability info with over-the-air update capability, or the like), an HMI to which gathers information from and to the driver or a passenger about the road ahead. The LDM may include a constantly updated map information containing advanced map information for the vehicles incoming to each road segment so that they are not fully surprised but informed of road situation with unexpected irregulates as an example. It may also, as suggested in this explanatory example, provide information where the road segment does not have an evacuation capable additional lane where incapable vehicle could find a evacuation area without affect other traffic user (which may otherwise result in unexpected rear end collision, since some driver reaching from behind fail to be attentive). For example, the HMI may be used in order to plan a trip, select a route, or the like, and to ask the driver whether a suggestion is accepted. This implies that the driver has probably seen where such a situation can happen and realized the danger, which increased their awareness of the situation, because they could face more risk if they ignored the notification and the vehicle system started an RMF in a road segment where "stop" is forbidden (e.g., inside a tunnel, on a bridge, on a bendy road, etc.) and they could also get a fine if the emergency stop was due to the driver's failure to properly and timely respond to the system notification to take over the driving.

[0169] At 73, it is estimated how long the driver needs for a possible transition to manual driving. This may have personal characteristic and further depends on the driver’s state, such as determined at 71, and / or an emotional state, cumulative fatigue, NDRA involvement, and the like, which is determined based on a passive or active monitoring, e.g., via a camera, a behavior analysis, or the like. The monitoring includes a utilization of at least one of a 2D / 3D camera, event triggered camera, an eye tracker, in-cabin position determination means, seat position analysis, body / posture tracking means, steering wheel position detection means, pedal input, proximity sensor, torque analysis, fatigue / drowsiness evaluation, monitoring of NDRA, and an application HMI requesting active feedback from the driver.

[0170] At 74, the HMI reminds the driver of increased risk (not only the naturally caused risk but including risk of possibly being imposed of penalties, especially in case of intentional repeated violations) and / or tries to increase the driver’s attention when, for example, fatigue or distraction is detected. For example, the HMI may remind or prompt the driver to take a rest.

[0171] The HMI is, in this embodiment, a graphic user interface (GUI) to visualize the risk (but may also include an audio interface), penalty, violation history, or the like. The HMI further includes means such that the driver can interact with the HMI, such as a touch panel, gesture recognition means, audio processing means, a steering wheel input, or the like. Also handheld applications may be used as HMI although it is preferable not be used as a sole means of interfacing.

[0172] At 75, the HMI generates an early reminder to the driver in order to feed / enrich / recover their situational awareness of the incoming road situation, e.g., based on LDM and over the air event update, if available via V2X (vehicle to environment). The HMI may generate the reminder, for example, based on an update or externally received information (e.g., via V2X). Additionally or alternatively, on-board sensors may be used which, for example, analyze traffic signs, detect other vehicles (or pedestrians), or the like advising the driver if any uncertainty or risk raise along the planned route. Hence, the automation system may work cooperatively with the navigation system having its trip LDM data continuously updated and thus, may try to provide a safe drive under automation and predict a point of time which may require a transition to manual driving, such that the driver may continue their NDRA until the transition is necessary but still be enriched with possibly missed information unless provided by this added HMI. This reminder or reminders may be missed by the driver if still far from reaching critical point.

[0173] Based on the driver’s response of the reminders, the personalized time budget is updated and renegotiated, which may lead to an increase or a decrease of the time budget. It should be noted that, in some cases, it may not be sensible to re-iterate 73 to 75, if the time budget is decreased or not increased enough, such that, after the re-negotiation, the notification escalation is initiated.

[0174] Generally, the system keeps monitoring and updating the changes and new events (such as laybys evacuation spots or lanes or refuge areas, traffic severity that may change over time, and the like, and keeps updating the time budget. For example, a road segment supposed to be MRM capable may require a short time budget, but the road segment traffic condition may change, such that a longer time budget might become necessary, if the road ahead MRM evacuation is not available anymore. Also, level 3 (conditional automation) may be downgraded to level 2 (partial automation) due to a sudden weather change, which may be recognized by the automation system.

[0175] At 76, a pre-notification is issued according to the situation and for further enhancing the situational awareness. For example, a loudspeaker, a haptic device on a seat, a seat belt, a wearable device, a handheld device, an illumination source, a graphical representation of CID (Center Information Display, which may also be configured referred as Instrument Panel (I / P) where speed meter and tell-tales are mandated by law and displayed), a prompt on a display, or the like may be used for the pre-notification. Hence, the pre-notification may include at least one of an earcon, a jingle, an audio signal, a haptic signal, a visual signal, a signal via a wearable or handheld device, a smell, or the like. The severity of the risk may be represented via the signal.

[0176] Moreover, the driver’s reaction to the notification is detected. The pre-notification is designed such that the driver perceives it as not annoying and is just used as a reminder. However, due to this nature, the pre-notification may be missed by the driver. In some embodiments, the prenotification may have different purposes, e.g. according to when they are issued. The main purpose is to inform the operator of the vehicle that there is a change in the steady stable state to be worth considering. This includes, for example, a detection of a traffic sign informing road work ahead which could be visually missed by the driver. In this case, the system may have triggered the pre-notification in advance to provide an adviser to the operator to check for the detected change(s) . This is the most common pre-notification which is used as a preparatory stage to notify driver that had there is change of driving condition ahead that requires driver intervention to be notified This allows that the driver could start recovering and preparing and, e.g. in the case that the driver is involved in a secondary task requiring additional time budget, such as closing an application on nomadic device inserting time consuming data which cannot be closed instantly, the system may adapt the timing of the pre-notification accordingly. This prenotification although not annoying by nature may serve as indicator for the operator to manage the remaining time how to terminate the secondary task involved. In case of getting involved in a time consuming transaction, the pre-notification timing could be combinedly determined to be provided with additional marginal timing so that the operator does not get panicked while in the middle of transaction resulting in unsafe transition to manual driving or MRM triggered due to failure to immediately stop a second task. Depending on the reaction, at 77, a notification is issued for initiation the transition, which is followed by a detection of the driver intent. The same or different means may be used as for issuing the pre-notification. The driver’s response may be detected, for example, by a 2D / 3D camera, an eye tracker, face analysis, pedal analysis, early foot / leg tracking (e.g., via a switch or proximity sensor). The reaction may lead to a suppression of a planned waming / alert.

[0177] Depending on the reaction, at 78, a warning and possibly an alert is issued that MRM / RMF will be initiated. As discussed herein, the alert and warnings may be designed such that the driver perceives them as more and more annoying, that is to say progressively changing. For example, more and more means for outputting a waming / alert may be used and the signals may be increased in intensity. In some embodiments, the vehicle automatically drives into an lay-bys evacuation spot / lane if it is detected that the driver reacts too late or not at all. As discussed herein, the driver behavior and the quality of the transition is recorded in a time-stamped manner for training an Al that predicts the driver’s possible behavior. The same raw data may also be maintained locally recorded. Additionally or alternatively a part of extracted data is recorded and stored in a separate memory area in an encrypted manner, such that it is easily retrievable according to specified procedure to be processed together with resulting situation as described at 79.

[0178] At 79, the transition (or the failure of the transition, i.e., whether MRM / RMF is activated) is evaluated and a score is generated which corresponds to a rewards or a penalty, as discussed herein. From the collected pre- and post-transition time stamped data from any of the output and input sources discussed herein, and possibly also vehicle control and stability data, the score is generated and the quality of the overall transition sequence is evaluated. For example, it may be evaluated that a smooth transition happened when a torque sensor (in a steering) wheel has a predetermined curvature. Also other sensors may be used, such yaw, pitch, roll sensors. The vehicle’s kinetic stability may be analyzed, which may be recorded on a DSSAD (Data Storage System for Automated Driving which stores incident event related data), or the like, and may interact with a classification tool with data encryption capability. The data encryption system and easy to retrieve function could be a part of the system or can be an isolated system so that personal privacy data could be managed and handled under driver own control.

[0179] At 79, time-stamped behavior records are generated for: generating / updating a personal response dictionary to be used in future time budget estimation. For example, an older driver might need a different time budget than a younger driver generating evidential data on possible violation, e.g., when the automation is used beyond allowance. The evidential data may be encrypted in according with data protection regulations, as discussed herein, but it may still be possible to retrieve the relevant data. Also, based on this data, a virtual risk may be generated for motivating the driver, as discussed herein.

[0180] Fig. 8 depicts an embodiment of a passenger safety method 80 according to the present disclosure.

[0181] At 81, an operator readiness level is determined based on an operator foot position, as discussed herein.

[0182] At 82, it is determined whether an operator warning is to be issued, as discussed herein.

[0183] At 83, the operator warning is issued or temporarily withhold based on the determined operator readiness level detected intent to intervene, as discussed herein.

[0184] Fig. 9 depicts an embodiment of a passenger safety method 90 according to the present disclosure in which databases are updated based on the operator’s reaction.

[0185] At 91, a foot displacement of the operator is detected between two different points of time for determining the operator readiness level, as discussed herein.

[0186] 92 to 94 basically correspond to 81 to 83, such that a repetitive description is omitted.

[0187] At 95, a warning database is updated based on whether a warning was issued, as discussed herein. For example, a database record is created or updated. Also, a motion database is updated for determining a future operator readiness level, as discussed herein. an operator readiness level is determined based on an operator foot position and motional nuance, as discussed herein.

[0188] At 82, it is determined whether an operator warning is to be issued, as discussed herein.

[0189] At 83, the operator warning is issued based on the determined operator readiness level, as discussed herein.

[0190] In some embodiments, a set of combined functions, which are described herein and that boost the acting and operand conditioning on driver behavior development, can be summarized as follows:

[0191] 1. Providing reward on pro-active response to system notification to intervene without delay,

[0192] 2. Avoiding inappropriate issuing of warning and alert whenever driver comes interacting promptly and properly to the system notification, thus preventing driver to get less respectful to warning as will be issued in adequately mostly when intention could not be detected in advance of warning issuing shallow time window

[0193] 3. Giving transparency to self-transition history with record of past transition, with visualized risk of getting interlocked or penalized due to repeated irregular use when violation registration is maintained registered

[0194] 4. Personal data encrypted in a proper manner under the controllability of driver for better user acceptance and system personalization to adaptively provide feedback, with extracted response behavior registered for rewarding / penalization, and transparency of the records inducing virtual and perceivable sense of risk; if not properly responding to the system request to intervene in a prompt manner, and being high exposed and prone to be post checked on any future traffic check point,

[0195] 5. Proactive responsive easily differentiated and discriminated by the use of foot initial motion detection to determine properly intention to intervene,

[0196] 6. Easy to retrieve and scroll back of driver record in order to check past driver record, which has been stored before stopping of the vehicle, at any near past point by traffic check point, enabling practical in an easy to scroll event records tool made available according to local traffic regulation

[0197] 7. Prioritization of the driver to be highly responsive in MRM prohibited road segment, but still permitting lower successful transition in accordance road acceptance of vehicle evacuation MRM

[0198] 8. Combining of these layered practices may allow a kind of psychological operand conditioning, where driver get self-educated according to repeated practice when the driver prefers not to endup having its automation function interlocked and disabled. This interlock may be a common function introduced to prevent driver of repeated misuse or abuse resulting in low quality transition or transition resulting in activation of MRM by respective regulation.

[0199] In the following, embodiments are discussed for detecting body features (such as a foot, ankle, leg, calf, thigh or the like, as discussed herein) of a passenger of a vehicle, e.g., for determining an operator’s intention to intervene according to the present disclosure. The following description is based on the application of time-of-flight (ToF) technology, but may be carried out likewise based a combination of 2D image sensor technologies, radio wave, millimeter radar sensor, IR imaging, NIR, imaging, a combination of NIR and RGB or any other combination, or the like, further including complementary use (with ToF) to achieve high precision operation and the overall energy efficiency.

[0200] Occupant monitoring in vehicles is generally known. It has been recognized that known approaches which include ToF for in-cabin monitoring may achieve inferior results compared to infrared (IR) alone. For example, IR imaging may have at least twice as good an SNR than ToF. Also, a contrast of IR imaging may be better. A spatial resolution of ToF camera may be lower than that of IR imaging.

[0201] However, it has been recognized that depth data or representations of depth provided by ToF may be useful for in-cabin monitoring.

[0202] Moreover, it has been recognized that an IR image may be obtained from a ToF measurement too, e.g., by using a single tap component of iToF as an approximation of the image content included in the IR image. To improve the SNR of the IR image, all taps and components of iToF may be added together. It has further been recognized that such information may be used for exposure control, e.g., based on a region of interest (ROI). For example, the single tap component may be used for exposure control since it may be indicative of ambient light.

[0203] It has further been recognized that it may be possible to improve the ToF confidence signal to get closer to IR imaging performance in terms of resolution. For example, spatial resolution may be increased with super resolution with two (or more) taps and depth information. For example, the increased spatial resolution may be used for more precise monitoring of body parts, such as eye ROI (which might not be possible with the original resolution).

[0204] Also, a modulation frequency may be tuned to get more precise depth and increase accuracy. A more accurate depth may be combined with confidence images for super resolution. Noise removal algorithms may be used to further increase the SNR in confidence images. Also, exposure and gain control may be used for comparative contrast in different ROIs.

[0205] It has been recognized that the following aspects may be useful to implement in order to increase a spatial resolution with two (or more) taps:

[0206] • Using confidence image with depth to detect salient features in the depth image out of which one can perform salient body segmentation, thereby obtaining reliable body segments present in the depth images.

[0207] • Using the confidence and clean depth for detection of a head of the occupant(s) along with other available body segments. • Using head IR information to adapt the gain to obtain higher dynamic range / contrast for confidence / IR component.

[0208] • Based on depth ROI, adapting a modulation frequency to get higher precision depth component for recognition of body part position for active / passive safety.

[0209] • The adapted depth and confidence / IR may then be used with segmentation to aid IR part to get higher resolution image of eye regions (super resolution with machine learning approach).

[0210] In such embodiments, as mentioned above, a machine-learning approach may be utilized, wherein the present disclosure is not limited to such embodiments.

[0211] In a non-machine learning approach according to the prior art, one would make a clean mesh, map confidence values, interpolate and then project the values to a 2D coordinate system. In such a case, there would be multi-samples (multi-tap in iToF) in time as well, which may be used for a 3D trajectory estimation which may then be filtered in time domain.

[0212] According to the present disclosure, higher depth accuracy may be achieved and may be used for salient features to detect a distance between an occupant to points of interest in the cabin.

[0213] Therefore, some embodiments pertain to time-of-flight circuitry for a vehicle safety system, the circuitry being configured to: obtain time-of-flight data indicative of an occupant of a vehicle; generate, based on the time-of-flight data, body feature information regarding the occupant; and provide the body feature information to a safety function of the vehicle safety system, e.g., adaptive restraint system.

[0214] The circuitry may be any entity or multitude of entities configurable to process time-of-flight data according to the present disclosure, such as a processor (CPU (central processing unit), GPU (graphics processing unit)), an FPGA (field-programmable gate array), a computer, a server, a camera, a driver monitoring system (DMS), or the like. Also, combinations of the above-mentioned entities may be envisaged, in some embodiments.

[0215] Time-of-flight data may be obtained based on a time-of-flight (ToF) measurement, such as in indirect measurement according to iToF (indirect time-of-flight), dToF (direct time-of-flight), spot ToF, or the like.

[0216] For example, the ToF circuitry may communicate with a ToF system or may be included in a system, such that the ToF circuitry may be configured to process the ToF data. For example, the ToF system may be positioned roughly around a rear-mirror of an in-cabin of the vehicle and thereby having a large field of view covering the whole cabin. It will be appreciated that some positions in the cabin and therefore the field of view may be occluded by certain fixed fittings, moveable fittings, occupants or articles introduced into the cabin by occupants (e.g., car fixtures, bag of an occupant, or the like), such that also embodiments may be envisaged in which the ToF system may be partly occluded.

[0217] Also, it should be noted that other positions of the ToF systems are envisaged according to the present disclosure depending on what ROI should be monitored, e.g., on a roof lining, a lighting console, behind a steering wheel, or the like.

[0218] The ToF data may be indicative of an occupant of a vehicle. For example, a corresponding ToF sensor / camera may be directed towards a position / location of an occupant (e.g., a driver) of the vehicle such that, when the occupant is in the vehicle, a ToF measurement may be used to acquire parameters (e.g., body features, as discussed herein) of an occupant.

[0219] Based on the ToF data, body feature information regarding the occupant may be generated. A body feature may include one or more body parts (such as for example a head, a forearm, a thigh, a torso, a hand, a forearm connected to a hand) that are recognized, a distance of a specific body part (e.g., from an exterior border of the body, such as a back of a head) to a location (or a point) of (or within) the vehicle, a posture of the occupant, body segmentation information, depth information of a body part, or the like. Body features may include different parameters such as upper body size of the occupant, width, age, gender, body volume, etc.

[0220] The body feature information may be generated based on a predetermined algorithm, which, for example, may be based on a data driven approach, such as an artificial intelligence, a neural network, a machine-learning approach, or the like. The predetermined algorithm may be a hard- coded algorithm, or it may include a learning algorithm which may further be developed / trained, e.g., when an (end) user uses the vehicle.

[0221] The body feature information may be provided to a safety function of a vehicle safety system, such as an Occupant Status Monitoring (OSM), adaptive restraint system or a driver monitoring system (DMS). The safety function may process the body feature information together with other information provided from different sensors, such as features from an RGB or IR camera, sound information from a microphone, temperature characteristics from a temperature sensor, breathing or pulse information for example from a radar sensor, or the like.

[0222] According to the present disclosure, a position of a body feature in steady state (e.g., a state in which no safety measure may be necessary) may be determined. Also a temporal dynamic behavior of the body feature(s) may be determined which may be indicative of information whether or not the occupant is responsive to a system notification (e.g., a warning), or whether the occupant returns to a nominal seating position from an irregular position.

[0223] However, such temporal behaviors of an occupant may be too abrupt to be captured in a rough detection and may require a precise capturing in high resolution as well as a high frame rate. According to the present disclosure, such predictive behavior of occupants may be achieved. However, such monitoring may have a high power consumption, such that it may not be operated continuously, in some embodiments, such that, in such embodiments, selective operation may be carried out based on a selected ROI, as described herein.

[0224] Accordingly, the ToF circuitry may be part of an OSM and / or DMS or it may be a separate unit in the vehicle.

[0225] In some embodiments, the body feature information is generated based on depth information of the time-of-flight data.

[0226] As generally known, ToF data may include depth information and confidence information. If (only) the depth information is used, a depth of different body parts of the occupant (with respect to the ToF sensor / camera) may be determined. Thereby, for example, a posture of the occupant may be determined.

[0227] Accordingly, in some embodiments, the body feature information is indicative of a posture of the occupant.

[0228] In some embodiments, the body feature information is generated based on confidence information of the time-of-flight data, such that segmentation of the occupant’s body may be carried out, as will be discussed further below.

[0229] Accordingly, in some embodiments, the body feature information includes body segmentation information of the occupant.

[0230] Both the depth and the confidence information may be used. Hence, in some embodiments, the body feature information is generated based on depth information of the ToF data and confidence information of the ToF data.

[0231] Although a distance between a predetermined body feature may be determined based on depth alone or confidence alone, it may be more effective (e.g., more exact) when both types of information are used.

[0232] Accordingly, in some embodiments, the body feature information is indicative of a distance between a body feature of the occupant and a predetermined location with respect to (or “of’ or “within”) the vehicle. Furthermore, according to the present disclosure, a predictive decisionmaking may be enabled causing a system to perform an adaptive safety control operation. Body features according to the present disclosure are expected to move smoothly and not likely to have an abrupt movement, in some embodiment, thereby enabling trajectory estimation. Also, a Kalman filter or the like may be provided to further improve occupant motion estimation.

[0233] Some embodiments pertain to a vehicle safety system including: time-of-flight circuitry configured to: obtain time-of-flight data indicative of an occupant of a vehicle; generate, based on the time-of-flight data, body feature information regarding the occupant; and provide the body feature information to a safety function of the vehicle safety system, as discussed herein.

[0234] The vehicle safety system may include or be combined with at least a part of an OSM, restraint system, DMS, or the like, as discussed herein.

[0235] In some embodiments, the safety function included in the vehicle safety system is configured to: obtain the body feature information.

[0236] In some embodiments, the safety function is configured to obtain intention estimation based on a temporal behavior analysis.

[0237] For example, by tracking a (detailed) behavior of an occupant, an anticipation / prediction of a body feature may be carried out before the occupant actually behaves according to the prediction. Hence, it may be possible to anticipate a position of a body feature before it reaches a destination area for activating a warning or an alarm, thereby enabling a counter-measure before it might be too late. Moreover, false alarms may be reduced.

[0238] In some embodiments, the safety function is further configured to: generate an alert based on a distance of a body feature of the occupant to a predetermined location with respect to the vehicle.

[0239] Some embodiments pertain to a method for time-of-flight circuitry for a vehicle safety system, the method including: obtaining time-of-flight data indicative of an occupant of a vehicle; generating, based on the time-of-flight data, body feature information regarding the occupant; and providing the body feature information to a safety function of the vehicle safety system, as discussed herein.

[0240] In some embodiments, body feature information is generated based on depth information of the time-of-flight data, as discussed herein. In some embodiments, the body feature information is indicative of a posture of the occupant, as discussed herein. In some embodiments, the body feature information is generated based on confidence information of the time-of-flight data, as discussed herein. In some embodiments, the body feature information includes body segmentation information of the occupant, as discussed herein. In some embodiments, the body feature information is generated based on depth information of the time-of-flight data and confidence information of the time-of-flight data, as discussed herein. In some embodiments, the body feature information is indicative of a distance between a body feature of the occupant and a predetermined location with respect to the vehicle.

[0241] Some embodiments pertain to a method for a vehicle safety system, the vehicle safety system including time-of-flight circuitry, the method including: obtaining time-of-flight data indicative of an occupant of a vehicle; generating, based on the time-of-flight data, body feature information regarding the occupant; and providing the body feature information to a safety function of the vehicle safety system, as discussed herein.

[0242] In some embodiments, the method further includes: obtaining, by the safety function, the body feature information, as discussed herein. In some embodiments, the safety function is further configured to: generating an alert based on a distance of a body feature of the occupant to a predetermined location with respect to the vehicle, as discussed herein.

[0243] From application performance perspective, the present disclosure may achieve precise tracking of an occupant’s behavior, e.g., by analyzing a temporal transition and further enabling to anticipate a near future position of a body feature. For example, based on an initial (small) motion that may be recognized, a prediction may be carried out, as discussed herein. Thereby, false alarms and incorrect operations due to inappropriate estimation may be reduced. If a system repeatedly sends false preventive alarm, for example to return to an appropriate seating position, the system may be perceived to be annoying to an occupant who may ignore the alarms.

[0244] Fig. 10 depicts an Occupant Status Monitoring (OSM) 100 as an embodiment of a vehicle safety system, which, in some embodiments, is also used as passenger safety circuitry according to the present disclosure.

[0245] The OSM 100 includes a ToF sensor 101, in this embodiment an iToF camera configured to acquire time-of-flight data indicative of a driver of a vehicle, without limiting the present disclosure in that regard since any other occupant may be monitored accordingly. The ToF data includes depth and confidence information as discussed herein.

[0246] The OSM 100 further includes ToF circuitry 102 according to the present disclosure.

[0247] Based on obtained ToF data, the ToF circuitry 102 is further configured to generate body segmentation information and to provide the body segmentation information to a safety function 103 included in the OSM 100. In this embodiment, the safety function 103 also obtains an RGB camera output 104 for the surveillance of the driver. It should be noted that the present disclosure is not limited to the case described herein since in some embodiments, an IR camera or RGB-IR camera may be used accordingly.

[0248] Fig. 11 depicts a schematic block diagram of a method 110 for generating body feature information 111.

[0249] ToF data including confidence 112 and depth 113 are fed into a confidence-based depth filtering algorithm 114 configured to generated different outputs, based on a coarse determination whether the obtained result has a sufficient quality:

[0250] One output is statistics for gain control 115 which is used for refining the confidence 112, i.e., a gain of the iToF camera is adapted based on this information. The confidence 112 is again used for refining the statistics for gain control 115.

[0251] Another output is statistics for modulation frequency 116 which is used to refine the depth measurement. For example, the modulation frequency may be adapted such that the depth measurement becomes more precise, as generally known to the skilled person.

[0252] Also, based on the confidence 112, segmentation labels 117 are generated based on a machinelearning algorithm. Based on the labels 117, the confidence 112, as well as the confidence filtered depth 114, the body feature information is generated. The labels, the confidence, and the depth may allow the system to estimate the 3D positions and orientations of predetermined body parts. Thereby, a corresponding skeleton fitting may be carried out based on which the body features are determined.

[0253] Fig. 12 depicts an illustrational result of a method according to Fig. 11. An interior of a vehicle is shown captured with an iToF camera, wherein the ToF data are only pre-processed at that point. In the interior, a driver 131 and a passenger 132 are shown.

[0254] Based on the pre-processed depth, certain points 133 (white rectangles) on the occupants are identified which are indicative of their respective postures (indicated with white lines in Fig. 12).

[0255] Based on the identified points, a distance to a predetermined location in the vehicle (or with respect to the vehicle) is determined. In this embodiment, a distance between the driver’s 131 head and a steering wheel 134 is determined. In some embodiments predetermined locations may be fixed positions, such as fixed armrests, corners of windows, or positions on the roof lining, where adaptive restraint system might be placed It should be noted that any distance between identified points and predetermined locations may be determined, in some embodiments. Also, multiple distances may be determined in some embodiments.

[0256] It should also be noted that a distance to any point (or location) of the vehicle may be determined and the present disclosure is not limited to points of the in-cabin. For example, a distance of a body feature to a windshield (inside or outside or intermediate point) may be determined. Also, a distance of a body feature between an intermediate layer of a car roof (e.g., a stabilizing metal layer) may be determined, in some embodiments.

[0257] Fig. 13 depicts a skeleton in a filtered depth image (top image) and the result of a segmentation is shown in a confidence image (bottom image). As in Fig. 12, certain points and a skeleton of the driver are identified in the depth image.

[0258] In the confidence image, body parts 141 are identified.

[0259] Similarly, Fig. 14 depicts a depth image (top left) and a confidence image (bottom left), wherein in the confidence image, points on the occupant’s body are identified based on the depth image with a high precision. Higher confidence values can be indicative of a front of the occupant’s body parts used for segmentation with depth information, while low confidence values indicate areas of depth with lower reliability to be discarded for further processing. For example, fingers of the driver can be recognized in this manner (indicated with white circles).

[0260] On the right of Fig. 14, it is shown that distances of the driver’s head and back are determined based on a method discussed herein and are used as an indicator of the driver’s position or posture. For example, in case it is determined, based on the determined distances, that the driver is bent forward, an alert is generated that he is out of position. Additionally or alternatively, an airbag restraint system is adapted or suppressed based on the driver’s position.

[0261] Fig. 15 depicts a method 150 according to the present disclosure in a block diagram. The method may be carried out by ToF circuitry according to the present disclosure and / or by a vehicle safety system according to the present disclosure.

[0262] At 151, ToF data are obtained, as discussed herein.

[0263] At 152, body feature information is generated and, at 153, provided to a safety function, as discussed herein.

[0264] Fig. 16 depicts a method 160 for a vehicle safety system according to the present disclosure in a block diagram.

[0265] At 161, body feature information is obtained, as discussed herein. At 162, an alert is generated based on a distance of a body feature of an occupant to a predetermined location with respect to the vehicle, as discussed herein.

[0266] In some embodiments, a method described under reference of Fig. 16 may further include learning (e.g., by repeated operation) the occupant’s personal behavior characteristics including his body posture and / or temporal dynamical movements indicative of whether the occupant is expected to move in order to determine whether an alarm and / or a further safety relevant countermeasure is required. A simple threshold-based detection operation is also envisaged, in some embodiments, but it may exhibit inferior performance to the adaptive operation explained herein.

[0267] It should be recognized that the embodiments describe methods with an exemplary ordering of method steps. The specific ordering of method steps is however given for illustrative purposes only and should not be construed as binding.

[0268] Please note that the division of the OSM 100 into units 101 to 103 is only made for illustration purposes and that the present disclosure is not limited to any specific division of functions in specific units. For instance, the OSM 100 could be implemented by a respective programmed processor, field programmable gate array (FPGA), or the like. It may further contain a personalization feature with a learning function and a dictionary, parametrization by individual users with exportable parameter sets, and the like.

[0269] Some embodiments pertain to a method for in-cabin monitoring (ICM) including at least one of the following aspects: obtaining wide field of view iToF images of people (e.g., occupants of a vehicle) and high confidence areas (where depth is reliable); fusing pre-processed (removed low confidence areas) iToF images; detection of occupants and measuring people parameters and their position in the vehicle; categorizing people and postures as a function of the measured parameters measuring the distance between specific body parts and the key in-cabin 3D points of interest; supplementing driver monitoring system related to head orientation and eye gaze

[0270] A wide field, as mentioned above, may be defined as an angle of view such that at least eighty percent of a width of a (front) cockpit area (e.g., bordered by the exterior dimensions of seats or by the windows, or the like) may be captured. The iToF images may be obtained from an iToF ICM sensor, wherein each image may correspond to a region of interest (ROI) of the iToF sensor. For example, different ROIs may have different resolutions, different frames per second (FPS), different size, or the like.

[0271] For example, in an ROI in which the head of the driver is included, higher resolution, FPS, or the like may be needed than in an ROI where no body part of the driver is identified. An ROI may be a rectangular region including the body part, but the present disclosure is not limited in that regard since any shape of the ROI may be envisaged. Also, it should be noted that an overlap of two body parts (of the same occupant or of different occupants) may be included in an ROI.

[0272] Accordingly, one or more ROIs may include a part or the whole head of the occupant with gain adaptation to obtain a higher dynamic range and / or contrast for confidence and / or IR component.

[0273] In some embodiments, one or more ROIs may include an eye region of an occupant with a resolution enhancement compared to other ROIs (e.g., super-resolution, such as two to four times higher). Such data may be used to supplement an OSM / DMS for head orientation and gaze monitoring.

[0274] For example, the super-resolution may be achieved with a machine-learning approach using a single reflection image and enhanced depth image.

[0275] In some embodiments, based on the ROI, a modulation frequency is adapted, thereby achieving a higher precision depth component for passive safety.

[0276] In some embodiments, an ROI includes a predetermined body region depth enhancement by use of an adaptive modulation frequency when a rough position and a 3D bounding box is assessed.

[0277] In some embodiments, an adapted depth and confidence / IR is used with segmentation to aid an IR part to get a higher resolution image of eye regions (super resolution with machine learning approach).

[0278] In some embodiments, a confidence image with depth to detect salient features in the depth image based on which salient body segmentation is performed, thereby obtaining reliable body segments present in the depth image.

[0279] In some embodiments, categories of people are used for occupant status monitoring (OSM).

[0280] In some embodiments, categories of people are used for passive safety features related to adaptive restraint systems. The categories may include a body size, such as small, medium, large, a gender, an age, and the like. Additionally or alternatively, weight may be a category. For example, a seating arrangement may adapt to an occupant’s weight, and so, known weight measurements may be re-purposed. Weight sensors may be used to determine where airbags should trigger, for example. Such weight measurements may be performed by using strain gauge layout under the occupant’s seat. Posture / position detection according to the present disclosure may additionally be used to determine whether it is safe to deploy an airbag or to suppress it depending on the detected occupant in irregular seating condition. However, the estimation based on strain gauge detection alone (without the determination according to the present disclosure) may have a limited detection accuracy and may not suffice to perform an intelligent adaptive airbag deployment control.

[0281] In some embodiments, besides parameters about the size, volume, weight, etc., seating positions, hands, leg, postures, and predicted movements based on intention detection as described herein may be envisaged.

[0282] In some embodiments, body segmentation may be carried out and a distance between specific segmentation body parts (head, chest, torso, or the like) to specific 3D points of interest, in vehicle for restraint system adaptation may be determined, e.g., to adapt a seat belt, airbag, or the like.

[0283] Assessing a driver’s or any other occupant’s physical status and its anticipated movements (or movement intention) may be further relevant when combined with the increased use of autonomous driving, e.g., when occupants depart from a traditional seating position while in drive and thus, in case of an accident, an inappropriate deployment of an airbag may cause severe injuries.

[0284] In some embodiments, posture may include at least one angle at which an occupant is seated, for example slouching or leaning towards the center console and armrest, or conversely towards the door, or twisting themselves around to look at what is happening in the passenger area or reaching for an object that has slipped under a seat.

[0285] Passive safety devices may be configured to act differently dependent on detected postures, e.g., based on a control signal. For example, airbags may inflate differently depending on the detected posture or behavior of the occupant. For example, airbags may contain different pockets or volumes which can be differently controlled, such as individually. For example, airbags may be inflated to different pressures. For example, airbags may be inflated from different inflation sources causing an airbag to be harder or softer on one side than another or fully inflated on one side before another is fully inflated.

[0286] Some safety features may, in the past, have been designed based on worst case scenario impacts with an emphasis on saving life. The disclosure envisages that safety features can be deployed in a manner such as to mitigate harm to the occupant. For example, it may not be appropriate to deploy an airbag in the same manner, if they are facing to the left, rather than facing forward. Therefore, an airbag may deploy in a first mode when the occupant’s head is detected to be facing in the direction of travel, and the airbag may deploy a second, different, mode when an occupant’s head is detected to be facing a direction which is greater than a first threshold angle. The airbag may deploy in a third mode, differently again, when an occupant’s head is detected to be facing a direction which is greater than a second threshold angle. It will be appreciated that head is merely an example of a body part and that the disclosure may apply to other body parts or combinations of multiple body parts. It will be appreciated that an airbag may be a system of multiple airbags or airbag pockets. The disclosure provides advantageous adaptability of safety features based on occupant posture.

[0287] It will be appreciated that the above description, for clarity, has described embodiments with reference to different functional units, circuitry and / or processors. However, it will be apparent that any suitable distribution of functionality between different functional units, circuitry and / or processors may be used without detracting from the embodiments.

[0288] Described embodiments may be implemented in any suitable form including hardware, software, firmware or any combination of these. Described embodiments may optionally be implemented at least partly as computer software running on one or more data processors and / or digital signal processors. The elements and components of any embodiment may be physically, functionally and logically implemented in any suitable way. Indeed, the functionality may be implemented in a single unit, in a plurality of units or as part of other functional units. As such, the disclosed embodiments may be implemented in a single unit or may be physically and functionally distributed between different units, circuitry and / or processors.

[0289] Although the present disclosure has been described in connection with some embodiments, it is not intended to be limited to the specific form set forth herein. Additionally, although a feature may appear to be described in connection with particular embodiments, one skilled in the art would recognize that various features of the described embodiments may be combined in any manner suitable to implement the technique.

[0290] Furthermore, the invention is described taking the conventional seating equipment as reference but it obviously could be further expanded to largely flexible equipment to be introduced in automated vehicles when driver and passenger are no longer restrained to a specific position within the vehicle, requiring adaptation of the detection to best fit these new needs. Some examples are reclined seat, turned seat accommodating the occupants, and the like. Fig. 17 depicts an embodiment of a method 170 according to the present disclosure for performing a steady state rough operation mode before defining a specific ROI and performing to a fine operation mode.

[0291] The operation of the device in high frame rate and high-resolution mode means the device needs to be operated at high operation frequency and therefore becomes less power efficient and dissipates a part of the energy as heat which may further deteriorate sensor performance due to noise induced by the temperature.

[0292] To save energy and suppress thermal noise, the device is operated at a moderated operation mode at a steady state operation mode (low frame rate, low resolution) when precise occupant evaluation is not required. In this steady state operation mode, it is still possible to perform occupant rough estimation based on an analysis of captured data.

[0293] At 171, the sensor is operated in nominal operation mode to capture the occupant’s status in rough mode and to check for irregularities, without running any further detailed analyses.

[0294] At 172, the captured data is analyzed to check whether the occupant might be out of an expected position and / or posture to self-trigger a need for detailed precision operation to estimate the precise driver status.

[0295] At 173, the system (OSM) further checks an external request flag whether or not to shift the operation to high precision (“FINE”) operation mode. The switching to this FINE operation mode can be determined by detecting “irregular occupant posture”, or the like, but as well taking into consideration the vehicle status while in drive (e.g., speed) and its risk prevailing with possibility for the need to deploy an airbag, or the like. For example, if the vehicle status is determined to be in an empty country road at low speed with negligible risk, it may not be necessary to assume that the airbag is required. Similarly, such an assumption may be made while maneuvering the vehicle in a garage or a parking spot (such that the airbag may be deactivated, in such cases). On the other hand, if a situation is identified in which an accident is imminent and the driver not seated in a secure seating position, an anticipation / prediction / estimation of a driver status may be required. In which condition set the device is required to switch to FINE precision operation could be parametrized to best fit the intended safety operation to the type of vehicle in interest.

[0296] At 174, the system makes the decision whether to shift the sensor operation to the FINE mode to enable better decision taking for the operation of passive safety system without harming the occupant with inappropriate deployment of the airbag (without limiting the present disclosure in that regard). If a FINE operation is required, it will exit the loop and go to a precise operation flow, such as described under reference of Fig. 18. If the shift of the operation mode to the precise (FINE) mode is not necessary, the method will maintain the operation loop to resample the driver status in the rough operation mode until the monitoring operation in no longer needed due to termination of the trip not illustrated in this flow.

[0297] Analyses and detection based on personalized learned characteristics are not given in this flow chart for simplicity, but it can well be adapted for rough occupant estimator. Operation flow described herein is an exemplary embodiment and it may be rearranged, e.g., by intermittent or occasional transition to fine operation to confirm driver ambiguously detected states, or the like.

[0298] There is a limited space available within a vehicle and thus, it may be necessary to minimize the number of equipment taking space of the passenger compartment. Therefore, in some embodiments, a driver status detector for passive safety is integrated or combined with equipment for the purpose of OSM / DMS.

[0299] Fig 18 provides an operation sequence flow chart when the FINE precision operation according to Fig. 17 is activated. According to the embodiment of Fig. 18, the same equipment is used for occupant behavior and intention estimation as the equipment used in Fig. 17 (thereby not occupying more space, as discussed above).

[0300] In this embodiment, an anticipation / prediction / estimation of the driver’s behavior is carried out based on an initial detection of movement. This is made possible by a personalized action classification based on historical data of the occupant that is used to learn a behavior dictionary with DNN machine learning (without limiting the present disclosure in that regard).

[0301] At 181, an operation control unit (not illustrated) determines an ROI for FINE precision tracking when indicated as described above. Also, other relevant parameters requiring precision detection are determined to enable improved high resolution and precise motion tracking required to achieve higher accuracy of estimation.

[0302] Once the operation of the image sensor is adapted to track a particular ROI with higher accuracy, at 182, it becomes possible to obtain posture details and precise motion enabling to conduct anticipation / prediction / estimation of the occupant’s future behavior by analyzing the driver’s time-sequential behavior with high accuracy. In other words, at 182, a detailed tracking is carried out to estimate an expected future motion (e.g., of at least one of body, upper posture, hand, foot, and the like).

[0303] Based on the anticipation / prediction / estimation, it can be determined that the occupant intends to return to a regular seating position, aborting any NDRA and returning his hands and feet back to their normal operating position, or the like, However, just tracking the motion does not necessarily provide the correct information necessary for a later system decision, e.g., the decision to emit an alarm, or the like.

[0304] At 183, the detected detailed tracking information is compared to the historical occupant behavior data to enable a behavior analysis based on personal behavior characteristics. For example, some drivers may be slow and other drivers may be quicker in reacting, thereby exhibiting different behaviors when compared to each other.

[0305] This evaluation is built upon cumulative personal behavior data analyses and based on a learning algorithm, stored and maintained as a baseline of a reference dictionary, which, in some embodiments may further include a health condition, a body feature, or the like, thereby adding biasing on a decision, e.g., based on a health factor (e.g., fatigue, drowsiness), and thus modifying the behavior characteristics.

[0306] At 184, the analysis system uses the detected motion estimation results to score the driver / occupant willingness to react by analyzing the detected motion and comparing it to the personal behavior tendency dictionary data.

[0307] The estimator uses the cumulative historical results learned based on the occupant’s historical record, as later described at 189. Thus, it is possible to predict the driver’ s / occupant’s expected behavior from the initial and detailed tracked cue, even when the driver / occupant has not reached an expected target position yet.

[0308] This anticipation / prediction is important since there may be a limited time budget in activating a pre-deployment system of an airbag, for example. In case of providing cues to the DMS / OSM system, the anticipation can improve the estimation of the driver’ s / occupant’s intention to intervene. If a decision is only made based on a final posture and / or position, there might be a chance that it is decided that an airbag is inflated too late or that it is not inflated at all due to an incorrect decision (or that it is not inflated although necessary).

[0309] In another example, if it fails to detect the willingness of the driver to take control of the vehicle in a critical situation, thereby causing the vehicle to initiate an evasive maneuver even in situation the driver is properly interacting with the system, a high risk of an accident may be at hand. Reducing false alarm or inappropriate intervention is crucial in achieving higher trust of the occupant in the system. Otherwise, the occupant may disable such a function.

[0310] At 185, the detected occupant / driver status is evaluated and the remaining time budget for taking a decision is estimated. For taking the decision, it may need to be balanced between remaining time versus running a further additional round of detailed occupant / driver tracking detection, as at 187. If not enough time is left for another round, it may be decided to directly start the preparation for the activation of an adaptive smart airbag according to the detected status or based on anticipated expectation of the targeted occupant position. In the case the detection results are fed to the OSM / DMS system, the detected results will serve to achieve improved accuracy on its estimation, especially in deriving a confidence level of the occupant’s intention to intervene.

[0311] At 187, another round of notification escalation is carried out, as discussed above, e.g., to find out whether the occupant reaches a safe seating position and / or to determine an optimal deployment of the safety system and / or to determine a counter-measure to be taken, if the driver does not intervene. Also, a notification is issued to the occupant / driver to return to a safe seating position. Also, in some embodiments, warnings may be escalated, if necessary (e.g., louder or more frequent) and the occupant’s behavior may be recorded and handed over to be stored as personal historical record for the learned occupant behavior dictionary, thereby updating the dictionary.

[0312] At 186, a decision is taken, as described above. If there is not enough time left, at 188, a passive safety system is started since a risk is too imminent.

[0313] It should be noted that if it is determined that no risk is imminent, the FINE mode is terminated and the ROUGH operation mode described in Fig. 17 is carried out.

[0314] The accumulated temporal behavior tracking data is analyzed and fed, at 189 into the learned occupant behavior dictionary.

[0315] The termination of the loop at 186, i.e., to move to 188 instead of repeating and proceeding to 187 is made when at least one of following conditions are met with further needs to continue the behavior monitoring of the occupant according to the prompt initiated at 171.

[0316] 1. Irreversible activation of the detonation of airbag determined

[0317] 2. Situation requiring pre-activation to prepare the airbag release re-establishing lock that avoids incorrect airbag detonation

[0318] 3. Occupant / driver back to the expected “rest” position detected and confirmed that the situation does no longer require further behavior tracking and release from alert state (in vehicle design, “rest” position (or another “irregular seating position) may be expected to not require an adaptive deployment of the airbag) 4. In operation functioning as a component of OSM / DMS to track driver behavior or the anticipated behavior detection, it may temporarily leave the FINE precision operation mode if MRM (minimum risk maneuver) operation is performed under safe conditions. However, termination of the loop does not apply if the MRM determines that there is an imminent risk during the MRM maneuver, occupant / driver behavior tracking continues to monitor and estimate the occupant / driver and provide necessary posture and related detail to prepare for the adaptive deployment of the airbag or otherwise reaches a safe stop without having to detonate the airbag, e.g., by independent triggering mechanism built in the airbag system itself.

[0319] Each of incident tracking of the detailed occupant behavior which had been temporally recorded stored as cumulated data is fed to behavior learning and used to refine the accuracy of the detector in 89 (occupant behavior dictionary) at exiting the loop.

[0320] It should be recognized that the embodiments describe methods with an exemplary ordering of method steps. The specific ordering of method steps is however given for illustrative purposes only and should not be construed as binding. For example the ordering of 4 and 5 in the embodiment of Fig. 1 may be exchanged. Also, the ordering of 21, 22 and 23 in the embodiment of Fig. 2 may be exchanged. Further, also the ordering of 31 to 34 in the embodiment of Fig. 3 may be exchanged. Other changes of the ordering of method steps may be apparent to the skilled person.

[0321] The methods discussed herein can also be implemented as a computer program causing a computer and / or a processor to perform the method, when being carried out on the computer and / or processor. In some embodiments, also a non-transitory computer-readable recording medium is provided that stores therein a computer program product, which, when executed by a processor, such as the processor described above, causes the method described to be performed.

[0322] All units and entities described in this specification and claimed in the appended claims can, if not stated otherwise, be implemented as integrated circuit logic, for example on a chip, and functionality provided by such units and entities can, if not stated otherwise, be implemented by software.

[0323] In so far as the embodiments of the disclosure described above are implemented, at least in part, using software-controlled data processing apparatus, it will be appreciated that a computer program providing such software control and a transmission, storage or other medium by which such a computer program is provided are envisaged as aspects of the present disclosure.

[0324] Note that the present technology can also be configured as described below. (1) Passenger safety circuitry for warning an operator of a vehicle with an autonomous driving function, the circuitry being configured to: determine an operator readiness level based on an operator foot position; determine whether an operator warning is to be issued; and based on the determined operator readiness level, issue the operator warning.

[0325] (2) The circuitry of (1), wherein the operator readiness level is determined based on the operator foot position at two different points of time.

[0326] (3) The circuitry of (2), further configured to: detect a foot displacement between the two different points of time, wherein the operator readiness level is determined based on the detected foot displacement.

[0327] (4) The circuitry of anyone of (1) to (3), wherein the operator foot position is determined based on a reference point in the vehicle.

[0328] (5) The circuitry of (4), wherein the reference point includes at least one of a pedal, a home position including a foot rest, a user defined position including at least one of a comfortable position and a normal position, a history-based position, and a position determined based on an artificial intelligence.

[0329] (6) The circuitry of anyone of (1) to (5), further configured to: issue the operator warning, if the readiness level is below a predetermined threshold.

[0330] (7) The circuitry of anyone of (1) to (6), wherein a type of the operator warning is based on the determined readiness level.

[0331] (8) The circuitry of anyone of (1) to (7), further configured to: update a warning database based on whether a warning was issued.

[0332] (9) The circuitry of anyone of (1) to (8), further configured to: update a motion database for determining a future operator readiness level.

[0333] (10) The circuitry of anyone of (1) to (9), wherein the operator readiness level is further based on an emotional state of the operator.

[0334] (11) A passenger safety method for warning an operator of a vehicle with an autonomous driving function, the method comprising: determining an operator readiness level based on an operator foot position; determining whether an operator warning is to be issued; and based on the determined operator readiness level, issuing the operator warning. (12) The method of (11), wherein the operator readiness level is determined based on the operator foot position at two different points of time.

[0335] (13) The method of (12), further comprising: detecting a foot displacement between the two different points of time, wherein the operator readiness level is determined based on the detected foot displacement.

[0336] (14) The method of anyone of (11) to (13), wherein the operator foot position is determined based on a reference point in the vehicle.

[0337] (15) The method of (14), wherein the reference point includes at least one of a pedal, a home position including a foot rest, a user defined position including at least one of a comfortable position and a normal position, a history-based position, and a position determined based on an artificial intelligence.

[0338] (16) The method of anyone of (11) to (15), further comprising: issuing the operator warning, if the readiness level is below a predetermined threshold.

[0339] (17) The method of anyone of (11) to (16), wherein a type of the operator warning is based on the determined readiness level.

[0340] (18) The method of anyone of (11) to (17), further comprising: updating a warning database based on whether a warning was issued.

[0341] (19) The method of anyone of (11) to (18), further comprising: updating a motion database for determining a future operator readiness level.

[0342] (20) The method of anyone of (11) to (19), wherein the operator readiness level is further based on an emotional state of the operator.

[0343] (21) A computer program comprising program code causing a computer to perform the method according to anyone of (11) to (20), when being carried out on a computer.

[0344] (22) A non-transitory computer-readable recording medium that stores therein a computer program product, which, when executed by a processor, causes the method according to anyone of (11) to (20) to be performed.

[0345] (23) Vehicle controller circuitry configured to: estimate, based on an artificial intelligence, an intention of an operator based on a detection of the operator’s behavior and to make a decision on whether the operator performs a motion indicating an intention to intervene; record the decision in combination to extracted data from the detected operator behavior; and provide a result (e.g., a parametrized result) of the intention to intervene for deciding whether a safety system needs to activate a safety function (e.g., evasive maneuver).

[0346] (24) Vehicle controller circuitry (e.g., such as of (23)) configured to: footwell space detection circuitry configured to detects an operator’s foot (e.g., in a rest or home position) in a pre-defined space adjacent to acceleration and brake pedal for determining whether the operator brings the foot on one of the acceleration and brake pedal, thereby determining whether the operator intends to intervene during automated driving.

[0347] (25) The vehicle controller circuitry of (23) or (24), further configured to: estimate an individual characteristic of the operator for discriminating between the intention to intervene and a normal foot movement.

[0348] (26) The vehicle controller circuitry of (25), further configured to estimate the operator intention based on other driver detector data, such as based on at least one of an eye tracker, a head position estimator, a body posture estimator, and the like.

[0349] (27) Vehicle controller circuitry (such as any of (23) to (26)) comprising: body feature tracking circuitry configured to track at least one body feature of an operator of a vehicle based on at least one of the following devices: two-dimensional camera, three- dimensional camera, infrared sensitive detector, and a vital wearable device; and a data recording system (e.g., non-volatile) whose data is encrypted and made retrievable based on a handheld driver intervention confirmation device.

[0350] (28) The vehicle controller circuitry of anyone of (23) to (27), wherein the footwell layout is such that the operator can place the foot on it in a comfortable manner, and / or wherein the footwell layout serves, in a (semi-)predefined position, as an indicator for the operator intention to keep the foot at rest during the operation under automation, and as an indicator whether the operator intends to change its rest status.

[0351] (29) The vehicle controller circuitry of anyone of (23) to (28), further configured to: learn an individual behavior characteristic for generating operator specific early intention estimation based on cumulative historical learning, thus enabling determination of an operator specific individual characteristic for determining whether the active foot (or leg) is in the rest position (and e.g., only pretends to react and intervene) or whether the operator intends to intervene. (30) A behavior tracking method comprising: tracking an operator intent of an operator of a vehicle; provide an incentive point, if the intent is to intervene; and delete a past reward or generate a penalty, if the operator does not intervene although it is determined that the operator should intervene.

[0352] (31) An analysis and decision-making device comprising at least one of: a data input element where sensor raw data or pre-processed data is captured for a learning procedure using an artificial intelligence; a data input element that receives and captures the resulting effects or decision taken as post decision results, where the resultant effects are classified as good results when the operator starts interacting with the automation to successful hand-over and as bad results when the operator poorly interacts with the automation system leading the system to initiate a safety countermeasure to mitigate an escalation to a dangerous situation; wherein an artificial intelligence (such as the Al mentioned before in (23)) uses the resultant effects as positive (good results) or negative (bad results) as an input for assessing an operator performance; a personal operator behavior dictionary; a data input element where the operator emotional factor is acquired as an influencing factor of the driver behavior as an additional attribute that affects the operator’s decision to interact, and wherein a time required for a successful take over after the emission of prenotification, a notification, and a warning is acquired.

Claims

CLAIMS1. Passenger safety circuitry for warning an operator of a vehicle with an autonomous driving function, the circuitry being configured to: determine an operator readiness level based on an operator foot position; determine whether an operator warning is to be issued; and based on the determined operator readiness level, issue the operator warning.

2. The circuitry of claim 1, wherein the operator readiness level is determined based on the operator foot position at two different points of time.

3. The circuitry of claim 2, further configured to: detect a foot displacement between the two different points of time, wherein the operator readiness level is determined based on the detected foot displacement.

4. The circuitry of claim 1, wherein the operator foot position is determined based on a reference point in the vehicle.

5. The circuitry of claim 4, wherein the reference point includes at least one of a pedal, a home position including a foot rest, a user defined position including at least one of a comfortable position and a normal position, a history-based position, and a position determined based on an artificial intelligence.

6. The circuitry of claim 1, further configured to: issue the operator warning, if the readiness level is below a predetermined threshold.

7. The circuitry of claim 1, wherein a type of the operator warning is based on the determined readiness level.

8. The circuitry of claim 1, further configured to: update a warning database based on whether a warning was issued.

9. The circuitry of claim 1, further configured to: update a motion database for determining a future operator readiness level.

10. The circuitry of claim 1, wherein the operator readiness level is further based on an emotional state of the operator.

11. A passenger safety method for warning an operator of a vehicle with an autonomous driving function, the method comprising: determining an operator readiness level based on an operator foot position; determining whether an operator warning is to be issued; andbased on the determined operator readiness level, issuing the operator warning.

12. The method of claim 11, wherein the operator readiness level is determined based on the operator foot position at two different points of time.

13. The method of claim 12, further comprising: detecting a foot displacement between the two different points of time, wherein the operator readiness level is determined based on the detected foot displacement.

14. The method of claim 11, wherein the operator foot position is determined based on a reference point in the vehicle.

15. The method of claim 14, wherein the reference point includes at least one of a pedal, a home position including a foot rest, a user defined position including at least one of a comfortable position and a normal position, a history-based position, and a position determined based on an artificial intelligence.

16. The method of claim 11, further comprising: issuing the operator warning, if the readiness level is below a predetermined threshold.

17. The method of claim 11, wherein a type of the operator warning is based on the determined readiness level.

18. The method of claim 11, further comprising: updating a warning database based on whether a warning was issued.

19. The method of claim 11, further comprising: updating a motion database for determining a future operator readiness level.

20. The method of claim 11, wherein the operator readiness level is further based on an emotional state of the operator.

Images