VEHICLE SYSTEMS FOR CONTEXT-DEPENDENT ASSESSMENT

Abstract

User-oriented driver assistance system, comprehensive: at least one vehicle sensor (60, 134); a hardware-based processing unit (106); and a non-volatile computer-readable storage device (104), comprising: an activity unit (320) which, when executed by the hardware-based processing unit (106), determines, based on context-dependent input information, at least one of an alarm assessment output and a scene perception output, wherein the context-dependent input information includes or is generated by the output of the vehicle sensor (60, 134); and an output structuring unit (330) which, when executed by the hardware-based processing unit (106), determines an action to be performed on a vehicle (10) based on at least one of the alarm assessment outputs and the scene perception output determined by the activity unit (320), wherein the activity unit (320), when executed by the hardware-based processing unit (106): receives or generates a standard object list that was created at least on the basis of the output of the vehicle sensor (60, 134); and adapts the standard object list based on the context-dependent input information and outputs a customized object list; and The scene perception output includes the adapted object list.

Description

TECHNICAL AREA

[0001] The present disclosure relates generally to vehicle driving systems and in particular to systems and methods for assessing a driving situation based on user context, driving context and / or environmental disorder. BACKGROUND

[0002] This section provides background information on the present disclosure, which is not necessarily the prior art.

[0003] Sensor output is increasingly used in modern driving operations. Information from cameras, e.g., radar and LiDAR, is made available to a driver or used by an autonomous driving system.

[0004] In various applications, sensor information is combined into a standard object list. This standard object list identifies, for example, nearby vehicles and pedestrians.

[0005] For each object in the standard object list, a Time-to-Collision (TTC) value can be determined, which only considers the dynamics - relative positioning and movement.

[0006] DE 10 2016 222 499 A1 discloses a method for operating a motor vehicle, wherein an operating device of the motor vehicle is configured to receive an operating action from a user and to implement it on the basis of a predetermined operating sequence protocol, which describes a sequence of control steps to be carried out by the operating device to generate an activation signal depending on the received operating action.

[0007] DE 10 2015 116 160 A1 discloses a head-up display unit for a vehicle for generating a virtual image of the vehicle's surroundings in the driver's field of vision. The display unit comprises an emission unit for generating and emitting an image signal that is projected onto a multiply folded projection path into the driver's field of vision. The display unit includes a movable reflector that influences part of the projection path, the movement of the emission unit being executed depending on a detected driving situation.

[0008] DE 10 2015 110 348 A1 discloses a device with a processing circuit designed to determine which of a plurality of optical warning devices is located in a first detected field of view of a driver; and to control a first optical warning device from the plurality of optical warning devices so that it issues a first optical warning when it is determined that the first optical warning device is located in the first detected field of view of the driver.

[0009] DE 10 2014 011 264 A1 discloses a method for calculating a recovery time required to direct the attention of a driver of a motor vehicle to the road traffic, wherein the driver is monitored by means of sensors and a state of attention of the driver is determined, wherein one or more values ​​are calculated from the determination of the state of attention, to which an expected recovery time is assigned. SUMMARY

[0010] Improved driver assistance is achieved through the use of data from the user context, the driving context and / or environmental disorder.

[0011] The present technology relates in various aspects to a user-centered driver assistance system for implementation in a transport vehicle. In various embodiments, the system includes a hardware-based processing unit and one or more vehicle sensors, such as a camera, radar, and LiDAR. The system also includes a non-volatile, computer-readable storage medium with an activity unit and an output structuring unit. When executed by the processing unit, the activity unit determines, based on context-dependent input information, at least one of the results of an alarm evaluation output and a scene perception output. The context-dependent input information is either contained within or generated from the output of the vehicle sensor.

[0012] The output structuring unit determines an action to be performed on the vehicle upon execution, based on at least one of the results of the alarm evaluation and the scene perception determined by the activity unit.

[0013] In various embodiments, the context-dependent input information includes the user context, which relates to a state or condition of a vehicle user.

[0014] The user's condition or state is determined in various embodiments based on (i) biometric data from user measurements, (ii) user activity, (iii) vehicle occupancy and / or (iv) user gaze.

[0015] In some cases, the state or condition is generated at least partially based on at least one user model.

[0016] The user model can be stored, for example, in the vehicle, on a mobile device, or on a remote server. The user model includes data indicating current or past actions, tendencies, reactions, or other interactions (with the vehicle or other stimuli), preferences, and other characteristics related to a presence, an object, a user, and / or other users. The user model data used can be current data received and considered in real time via the vehicle system, and / or historical data about the respective user and / or other vehicle occupants.

[0017] Historical data can be used, for example, as a basis for adjusting or interpreting current and / or past actions or interactions with a particular user. The use of historical data is described in more detail below.

[0018] In various embodiments, the context-dependent input information includes the driving context, which relates to the current driving conditions of the vehicle.

[0019] The context-dependent input information includes clutter information, which relates to a set of visual elements in an environment outside the vehicle that the vehicle sensor or the user can perceive in various embodiments.

[0020] In various embodiments, the alarm evaluation output includes an alarm evaluation.

[0021] The activity unit, when executed by the hardware-based processing unit, (i) receives or generates a default object list created at least on the basis of the vehicle sensor output, and (ii) adapts the default object list on the basis of context-dependent input information and outputs an adapted object list, with the scene perception output including the adapted object list.

[0022] In various embodiments, the action determined by the output structuring unit includes controlling (i) a time at which a vehicle-related notification is issued via a vehicle output device, (ii) a substance of the vehicle-related notification, (iii) a modality or modalities for the notification, and (iv) an amount or type of highlighting for the notification.

[0023] The action determined by the output structuring unit includes, in various embodiments, the control of an automated driving component of the vehicle.

[0024] In various embodiments, the action determined by the output structuring unit includes generating communication for transfer to a target computer device that is not part of the vehicle.

[0025] Various aspects of the present technology include non-volatile, computer-readable storage devices, processing units, and algorithms configured to perform any of the described operations.

[0026] Further aspects of the present technology will be partly explained below and partly explicitly mentioned. DESCRIPTION OF THE DRAWINGS Fig. Figure 1 schematically illustrates an exemplary transport vehicle with local and remote computer devices according to embodiments of the present technology. Fig. Figure 2 schematically illustrates some details of an exemplary vehicle computer from Fig. 1 in conjunction with local and remote computer devices. Fig. Figure 3 shows another view of the vehicle, highlighting exemplary storage components including system units. Fig. 4 shows the interaction between the components of Fig. 3, including external devices.

[0027] The figures are not necessarily to scale and some features may be enlarged or reduced to illustrate the details of certain components. DETAILED DESCRIPTION

[0028] Detailed embodiments of the present disclosure are disclosed herein as required. The disclosed embodiments serve only as examples, which can be implemented in various alternative forms and combinations thereof. As used herein, for example, exemplary and similar terms refer to embodiments that serve as an illustration, a specimen, a model, or a sample.

[0029] In some cases, known components, systems, materials, or processes have not been described in detail to avoid obscuring the present disclosure. Consequently, the disclosed structural and functional details are not to be understood as limitations, but merely as a basis for the claims and as a representative framework to convey to those skilled in the art the various possible applications of the present disclosure. I. Technology Introduction

[0030] The present disclosure describes various embodiments, systems and methods for assessing a driving situation based on user context, driving context and / or environmental disorder.

[0031] User context information indicates one or more factors that influence a user's ability to make decisions related to driving or operating the vehicle. The term "user" includes a person fully operating the vehicle, a person partially operating the vehicle with semi-autonomous driving modes, or a person using fully autonomous driving. The term "driver" can be used interchangeably with "user" here and can therefore also refer to a person using fully autonomous functions. The term "driver" can also refer to one or more relevant vehicle systems in implementations where the vehicle performs partially or fully autonomous functions.For example, if a component or unit of the present technology issues a notification of an imminent threat—such as a vehicle nearby or a large pothole—the notification can be transmitted to the driver or user, who may be a human operator, a human user (e.g., a person taking control of the vehicle or giving instructions to the autonomously driving vehicle), and / or one or more relevant vehicle systems.

[0032] In general, user context information includes user activities or similar situations that partially distract the user from the driving scene, regardless of whether the user is partially, fully, or not at all (e.g., during autonomous driving) driving the vehicle at that time. An example of user context includes a user participating in a phone conversation or otherwise using a personal device or gadget—for example, a book. Another example is user context where other people are in the vehicle and potentially any interactions between them and the driver, such as a small child who is present and crying, or an adult who is present and talking to the driver.

[0033] In various implementations, the user context also includes the user's mental or other physical characteristics. For example, the system might consider a known visual or hearing impairment as part of the user context. Another example is that the user context could include known or specific slow or impaired user reflexes, or the ability to respond to stimuli such as a required driving maneuver or a prompt that encourages the user to make a decision related to autonomous driving.

[0034] In various implementations, the user context can also include settings or parameters of the user model. The system can be configured to set actual values ​​or other data, or estimated or predicted values ​​related to the user—such as values ​​or data indicating the user's level of attention, whether the user notices an object in the driving scene, and so on. In various implementations, values ​​or data are estimated or predicted with an acceptable degree of uncertainty, but the system can determine whether the values ​​or data have at least sufficient certainty before using them.

[0035] Other inputs to decision units relating to the environment and driving conditions, such as environmental clutter or road or weather conditions, are integrated into the user context in some embodiments.

[0036] In various configurations, the system uses user-oriented hazard assessments or alarm assessments to determine a value or score. The user-oriented alarm assessment value can then be used by the system or other devices—such as nearby vehicles or pedestrian telephones—to determine the next course of action.

[0037] Further actions may include, for example, controlling the vehicle, initiating communication with the user, and initiating control of the operation or communication to (and possibly to and from) the other device.

[0038] The term environmental clutter generally refers to scenarios where the environment surrounding the vehicle, as detected by one or more vehicle sensors, contains a relatively high amount of visual information that the user must process when assessing the environment for driving purposes. A scene around a vehicle driving or being driven in Times Square, New York, with its many signs, pedestrians, taxis, and other vehicles, would typically have more environmental clutter than, for example, a scene during a drive in the countryside.

[0039] Clutter information in various implementations includes a variety of relevant objects—such as other vehicles, pedestrians, and potholes—as seen, for example, within a field of view. In planned implementations, clutter information can also indicate other qualities of the object, such as its position. Clutter information can include data showing object clustering, the locations of objects or clusters, and distinctions indicating which objects or clusters are more threatening or important, such as being in or predicted to be in a vehicle's path, compared to objects expected to remain adjacent to the vehicle's path.

[0040] The clutter measure is used in various forms to estimate the probability that the driver will perceive one or more objects in the driving environment. Generally, the probability of the driver noticing an object decreases with greater clutter. This probability is used to determine an alarm strategy to inform the driver about the object(s).

[0041] While selected examples of the present technology describe transport vehicles or modes of travel, and in particular automobiles, the technology is not limited by this focus. The concepts can be extended to a wide variety of systems and devices, such as other transport or moving vehicles, including aircraft, watercraft, trucks, buses, wagons, trains, commercial or manufacturing equipment (e.g., forklifts), construction and agricultural machinery, storage facilities, household appliances, personal or mobile computing devices, and the like. II. Carrier vehicle - Figure 1

[0042] Turning now to the characters, and especially the first character, it shows Fig. 1 an exemplary receiving structure or device 10 in the form of a vehicle.

[0043] The vehicle 10 includes a hardware-based control system 20. The hardware-based control system 20 includes a communication subsystem 30 for communication with mobile or local computer devices 34, and external networks 40.

[0044] Through external networks 40, such as the Internet, a local network, a mobile or satellite network, vehicle-to-vehicle, pedestrian-to-vehicle or other infrastructure communication, etc., the vehicle can reach 10 mobile or local systems 34, or remote systems 50, such as remote servers.

[0045] Mobile or local devices 34 include, for example, but are not limited to, a user smartphone 31, a first exemplary wearable user device 32 in the form of a smartwatch, and a second exemplary wearable user device 33 in the form of a tablet. Other exemplary wearable devices 32, 33 include smart clothing, such as a shirt or belt, accessories, such as bracelets, or stylish jewelry, such as earrings, necklaces, and lanyards.

[0046] Another example of a mobile or local device is a user plug-in device, such as a USB mass storage device or a similar device configured to communicate wirelessly.

[0047] Another example of a mobile or local device is an on-board device (OBD) (not shown in detail), such as a brake sensor, accelerometer, rotor wear sensor, throttle position sensor, steering angle sensor, RPM gauge, brake torque sensor, or other vehicle condition or dynamics sensors retrofitted to the vehicle after manufacture. The OBD(s) may include or be part of the sensor subsystem referred to below by reference numeral 60.

[0048] The vehicle control system 20, which in intended embodiments includes one or more microcontrollers, can communicate via a CAN Controller Area Network and / or Ethernet. The CAN / Ethernet message-based protocol is typically designed for multiplexed electrical wiring with automobiles, and the CAN / Ethernet infrastructure may include a CAN / Ethernet bus. The OBD may also be referred to as a vehicle CAN / Ethernet interface (VCI, VEI, etc.) component or product, and the signals transmitted by the CAN / Ethernet may be referred to as CAN / Ethernet signals. In other embodiments, communication between the OBD(s) and the primary controller or microcontroller 20 is performed via a similar or different message-based protocol.

[0049] The vehicle 10 also features various mounting structures 35. The mounting structures 35 include a steering wheel, a center console, and a dashboard or instrument panel. The mounting structure 35 also includes or houses a plug-in connector 36 – for example, a USB connector – and a visual display 37, such as a touch-sensitive, input / output human-machine interface (HMI).

[0050] Vehicle 10 also features a sensor subsystem with sensors that provide information to control unit 20. The sensor input to control unit 20 is schematically located on the right side under the hood. Fig. Figure 1 shows exemplary sensors with base numbers 60 - 601, 602, etc.

[0051] The sensor data relate to features such as vehicle operation, vehicle position and vehicle posture, user features such as biometrics or physiological measures, and environmental features relating to the vehicle interior or the exterior of the vehicle 10.

[0052] Exemplary sensors include a camera 601 positioned in a rearview mirror of the vehicle 10, a dome or ceiling camera 602 positioned in a header of the vehicle 10, an environment-facing camera 603 (facing away from the vehicle 10), and an environment-facing area sensor 604. Inter-vehicle focused sensors 601, 602, such as cameras and microphones, are configured to detect the presence of people, activities, or other cabin activities or features. The sensors can also be used for authentication purposes in a registration or re-registration routine. This subset of sensors is described in more detail below.

[0053] Environment-facing sensors 603, 604 Detection features about an environment 11 include, for example, posters, buildings, other vehicles, traffic signs, traffic lights, road conditions, obstacles, pedestrians, etc.

[0054] The aforementioned OBDs can be considered as local devices, sensors of subsystem 60, or both in various embodiments.

[0055] Local devices 34 – for example, a user's telephone, user's wearable device, or user's plug-in device – can also be considered sensors 60, as in embodiments where the vehicle 10 uses data provided by the local device based on the output of a local device sensor. The vehicle system can use data from a user's smartphone, for example, displaying physiological data captured by a biometric sensor in the phone.

[0056] The vehicle 10 also includes cabin output components 70, such as audio speakers 701 and an instrument panel or display 702. The output components may also include a dashboard or center console display screen 703, a rearview mirror screen 704 (for displaying the image captured by a vehicle rear / backup camera), any vehicle vision device 37, autonomous driving components (steering, braking, etc.), and control components for controlling these components. III. On-Board Computing Architecture - Figure 2

[0057] Fig. Figure 2 illustrates in more detail the hardware-based computing or control system 20 of Fig. 1. The control system 20 may be referred to by other terms, such as a computing device, a control unit, a control device or a similar descriptive term, and may be one or more microcontrollers, as mentioned above.

[0058] The control system 20 is, in various embodiments, part of the aforementioned larger system 10, such as a vehicle.

[0059] The control system 20 includes a hardware-based computer-readable storage medium or data storage device 104 and a hardware-based processing unit 106. The processing unit 106 is connected or connectable to the computer-readable storage device 104 via a communication link 108, such as a computer bus or wireless components.

[0060] The processing unit 106 may be referenced by other names, such as processor, processing hardware unit, the like, or others.

[0061] The Processing Unit 106 can contain or consist of multiple processors, including distributed or parallel processors in a single machine or multiple machines. The Processing Unit 106 can be used to support a virtual processing environment.

[0062] The processing unit 106 may, for example, include a state machine, an application-specific integrated circuit (ASIC), or a programmable gate array (PGA) with a field-programmable gate array (FPGA). References herein to the processing unit that executes code or instructions to perform operations, actions, tasks, functions, steps, or the like could include the processing unit that directly facilitates, directs, or cooperates with another device or component to perform the operations.

[0063] In various embodiments, the data storage device 104 is any volatile medium, a non-volatile medium, a removable medium and a non-removable medium.

[0064] The term "computer-readable media" and variations thereof, as used in the description and claims, refer to physical storage media. The medium can be a device and can be non-volatile.

[0065] In some embodiments, the storage media include volatile and / or non-volatile, removable and / or non-removable media, such as main memory (RAM), read-only memory (ROM), electrically erasable and programmable read-only memory (EEPROM), solid-state memory or other storage technology, CD-ROM, DVD, BLU-RAY or other optical disc storage media, magnetic tape, magnetic disk storage media or other magnetic storage devices.

[0066] The data storage device 104 includes one or more storage units or modules 110 that store computer-readable code or instructions executable by the processing unit 106 to perform the functions of the control system 20 described herein. The units or modules and functions are further described below in connection with the Fig. 3 and Fig. 4 described.

[0067] In some embodiments, the data storage device 104 also includes auxiliary or support components 112, such as additional software and / or data support services of the processes of the present disclosure, such as one or more user profiles or a group of default and / or user-set preferences.

[0068] As intended, the control system 20 also includes a communication subsystem 30 for communicating with local and external devices and networks 34, 40, 50. The communication subsystem 30, in various embodiments, includes any wired input / output (I / O) 116, at least one wireless remote receiver 118, and one or more short- and / or medium-range radio transmitters 120. Component 122 is shown by way of example to emphasize that the system is configured to accommodate one or more other types of wired or wireless connections.

[0069] The long-range receiver 118 is configured in some embodiments to facilitate communication between the control system 20 and a satellite and / or a mobile network, which can also be referred to schematically by reference numeral 40.

[0070] The short- or medium-range receiver 120 is configured to facilitate short- or medium-range communication, such as communication with other vehicles, in vehicle-to-vehicle (V2V) communications, and communication with the traffic system infrastructure (V2I). More broadly, vehicle-to-entity (V2X) can refer to short-range communications with any type of external entity (e.g., devices connected to pedestrians or cyclists, etc.).

[0071] For V2V, V2I communication, or to communicate with other extra-vehicle devices, such as local communication routers, etc., the 120 short- or medium-range communication receiver can be configured to communicate using one or more short- or medium-range communication protocols. Example protocols include short-range communication (DSRC), Wi-Fi®, BLUETOOTH®, infrared, infrared data association (IRDA), near-field communication (NFC), the like, or improvements thereof (Wi-Fi is a registered trademark of the Wi-Fi Alliance, Austin, Texas; BLUETOOTH is a registered trademark of Bluetooth SIG, Inc., Bellevue, Washington).

[0072] Through short-, medium- and / or long-range wireless communication, the control system 20 can send information about the operation of the processor 106, such as messages or packetized data, to and from one or more communication networks 40.

[0073] The remote devices 50, with which the subsystem 30 communicates, are in various embodiments located near the vehicle 10, remote from the vehicle, or both.

[0074] The remote devices 50 can be configured with any suitable structure to perform the operations described herein. The exemplary structure includes any or all structures as described in connection with the vehicle computing device 20. For example, a remote device 50 includes a processing unit, a storage medium with units or modules, a communication bus, and an input / output communication structure. These functions are referred to as the remote device 50 in Fig. 1 and the cross-references provided by this paragraph.

[0075] While local devices 34 within the vehicle 10 in the Fig. 1 and Fig. As shown in the 2, each of them can be outside the vehicle and in external connection with the vehicle.

[0076] Exemplary remote systems 50 include a remote server (for example, an application server) or a remote data, customer service, and / or control center. A user computing or electronic device 34, such as a smartphone, may also be located away from the vehicle 10 and communicate with the subsystem 30, for example, via the internet or another communication network 40.

[0077] An example of a control center is the OnStar® Control Center, which offers features for interacting with vehicles and users, whether through the vehicle or via other means (such as a mobile phone), using long-range communication such as satellite or cellular connections. ONSTAR is a registered trademark of OnStar Corporation, a subsidiary of General Motors Company.

[0078] As mentioned, the vehicle 10 also includes a sensor subsystem 60, which comprises sensors that provide the control system 20 with information about objects, such as vehicle operations, vehicle position, vehicle orientation, user characteristics such as biometrics or physiological measures, and / or the environment around the vehicle 10. The arrangement can be configured such that the control system 20 communicates with or at least receives signals from sensors of the sensor subsystem 60 via wired or short-range wireless communication links 116, 120.

[0079] In various embodiments, the sensor subsystem 60 includes at least one camera and at least one area sensor 604, such as radar or sonar, directed away from the vehicle, for example to support autonomous driving. In some embodiments, a camera is used to detect the area.

[0080] Visual-light cameras 603 facing away from the vehicle 10 may include a monocular, forward-looking camera, such as those used in lane departure warning (LDW) systems. Other embodiments may include other camera technologies, such as a stereo camera or a trifocal camera.

[0081] Sensors configured to detect external conditions can be arranged or oriented in a variety of directions without deviating from the scope of this disclosure. For example, the cameras 603 and the area sensor 604 can be oriented at any position or selected position (i) forward from a front center point of the vehicle 10, (ii) rearward from a rear center point of the vehicle 10, (iii) laterally from a side position of the vehicle 10, and / or (iv) between these directions, in each case at or toward any elevation.

[0082] The 604 range sensor can, for example, include any short-range radar (SRR), ultrasonic sensor, long-range radar such as those used in autonomous or adaptive cruise control (ACC) systems, sonar, or a light detection and ranging (LiDAR) sensor.

[0083] Other exemplary sensor subsystems 60 include the aforementioned interior sensors (601, 602, etc.) that are configured and arranged (for example, positioned and installed in the vehicle) to detect activity, persons, interior conditions, or other features relating to the interior of the vehicle. Exemplary interior sensors (601, 602, etc.) include microphones, in-vehicle visual light cameras, seat weight sensors, user salinity, retinal or other user features, biometrics or physiological measures, and / or the environment surrounding the vehicle 10.

[0084] The interior sensors (601, 602, etc.) of the vehicle sensors 60 can include one or more temperature-sensitive cameras (for example, 3D, RGB, RGB-D, infrared, or thermography) or sensors. In various embodiments, cameras are preferably positioned at a high location within the vehicle 10. Exemplary positions include a rearview mirror and the overhead compartment.

[0085] A higher positioning reduces the interference from lateral obstacles, such as front seatbacks, which block second- or third-row passengers, or block more of them. A higher-positioned camera (light-based (e.g., RGB, RGB-D, 3D, thermal, or infrared)) or other sensor will likely be able to detect the temperature of each passenger's body—for example, torso, legs, and feet.

[0086] Two exemplary positions for the camera(s) are in Fig. 1 with the reference number 601, 602 etc. - indicated on the rearview mirror and one on the vehicle header.

[0087] Other sensor subsystems 60 include dynamic vehicle sensors 134, such as an inertial measurement unit (IMU) with one or more accelerometers, a wheel sensor or a sensor associated with the vehicle's steering system (e.g. steering wheel) 10.

[0088] The Sensors 60 can include any sensor for measuring a vehicle position or other dynamics, such as position, speed, acceleration or height - for example, a vehicle height sensor.

[0089] The sensors 60 can include any known sensor for measuring the vehicle's environment, including those mentioned above and others, such as a precipitation sensor to detect whether and how much it is raining or snowing, a temperature sensor, and others.

[0090] Sensors for capturing user characteristics include any biometric or physiological sensor, such as a camera for retinal or eye detection, facial recognition or fingerprint recognition, a thermal sensor, a microphone used for voice or other user identification, other types of user-identifying camera-based systems, a weight sensor, breath quality sensors (e.g., breathalyzers), a user temperature sensor, an electrocardiogram (ECG) sensor, electrodermal activity (EDA) or galvanic skin response (GSR) sensors, blood volume flow (BVP) sensors, heart rate (HR) sensors, electroencephalogram (EEG) sensor, electromyography (EMG), a salinity measurement sensor, the like, or others.

[0091] User vehicle interfaces, such as a touch-sensitive display 37, buttons, knobs, the like or others, can also be considered as part of the sensor subsystem 60.

[0092] Fig. Figure 2 also shows the cabin output components 70 mentioned above. The output components in various embodiments include a mechanism for communicating with vehicle occupants. The components include, but are not limited to, audio speakers 140, visual displays 142, such as the instrument panel, center console display screen, and rearview mirror screen, and haptic outputs 144, such as the steering wheel or seat vibration actuators. The fourth element 146 in this section 70 is provided to emphasize that the vehicle may include a variety of other output components, such as vehicle control components, for example, autonomous steering, brakes, or other autonomous controls or components. IV. Additional vehicle components - Figure 3

[0093] Fig. Figure 3 shows an alternative view 300 of vehicle 10 of the Fig. 1 and Fig. 2, which highlights exemplary storage components and shows associated devices.

[0094] As mentioned, the data storage device 104 includes one or more units and modules 110 for carrying out the processes of this disclosure. The device 104 may also include cross-process components 112, such as additional software and / or data that support the execution of the processes of this disclosure. These cross-process components 112 may, for example, be additional software and / or data that support the execution of the processes of this disclosure, such as one or more user profiles or a set of default and / or user settings.

[0095] Examples of user profile data include data received or generated by the vehicle processor that indicate the user's characteristics, such as whether the driver is temporarily impaired or has some type of disability that affects driving, even if minor, such as impaired vision or hearing. Characteristics can be displayed, for example, using data received from a remote server.

[0096] Each of the described codes or instructions can be part of more than one unit or module. And all the functions described herein can be performed by executing instructions in one or more modules, although the functions can be primarily described in connection with a unit or module using a primary example. Each of the units, sub-units, modules, or submodules can be designated by any number of names, such as a term or phrase that indicates its function.

[0097] Subunits or submodules can cause the hardware-based processing unit to execute specific operations or routines of functions of the units or modules. Each subunit or submodule can also be designated by any number of names, such as a term or phrase that indicates its function.

[0098] Exemplary units 110 and constituent subunits include: - Input unit 310 ◯ an optical disorder subunit 312; ◯ a subunit of the driver context 314; ◯ a subunit of the driving context 316; - Activity unit 320 ◯ a user-oriented hazard or warning, assessment subunit 322; ◯ a unit for estimating scene perception 324; - Output structuring unit 330 ◯ Threat assessment or alert assessment, output subunit 332, which may, for example, include an automated decision-making subsystem; ◯ Scene assessment subunit 334, which may include the same or a separate automated decision-making subsystem; and - Implementation unit 340 ◯ an output interface sub-unit 342.

[0099] Other vehicle components that are in Fig. Figure 3 shows the vehicle communication subsystem 30 and the vehicle sensor subsystem 60. These subsystems act, at least partially, as input sources to the units 110 and, in particular, to the input interface submodule 312.

[0100] Exemplary inputs from communication subsystem 30 include user identification data indicating user qualities such as age, gender, mood, attention capacity or limitation, hearing or visual impairment, the like, or similar.

[0101] The factor can be provided according to any suitable format, scale, or protocol. For example, attention capacity can be provided on a scale of 1 to 10. A mood factor can also be given on a scale or by categories—e.g., normal, tired, worried, annoyed, anxious, etc.

[0102] The communication subsystem 30 receives and provides data to the input unit 410 from any sources, including sources separate from the vehicle 10, such as local or mobile devices 34, including devices of pedestrians, other vehicle systems, local infrastructure (local stations, cell towers, etc.), satellite systems, and remote systems 50, which provide a variety of information, such as user identification data, user history data, user selections, or user preference context data (weather, road conditions, navigation, etc.), program or system updates. Remote systems may include, for example, application servers corresponding to the applications operating on the vehicle 10 and all relevant user devices 34, a user's or supervisor's computer (parent, job manager), vehicle dealers (e.g.,Service department), vehicle operator, customer control center system, such as the aforementioned OnStar® control center or a vehicle operator system, such as that of a taxi company that operates a fleet to which vehicle 10 belongs, or of a ride distribution service operator.

[0103] Exemplary inputs from the vehicle sensor subsystem 60 include, but are not limited to: - Environmental sensors that collect data about the conditions of a vehicle, such as from an external camera, distance sensors (for example, LiDAR, radar), as well as temperature sensors, precipitation or humidity sensors, or any sensor for detecting or measuring properties of the vehicle's environment. - Vehicle dynamics sensors, such as a vehicle speed sensor (for example, a tire rotation sensor), vehicle acceleration or other motion, such as an inertial impulse unit (IMU) with one or more accelerometers, vehicle position or other dynamics, such as position, speed, acceleration or height - for example, vehicle height sensor, brake sensors, steering angle sensor, all other sensors for providing vehicle gauges or telematics information; - Biometric / physiological sensors: Provision of biometric data about vehicle occupants, such as facial features, speech recognition, heart rate, salinity, skin or body temperature for each occupant, etc.; - Vehicle occupant input devices, such as human-machine interfaces (HMIs) for vehicles, such as a touch-sensitive screen, buttons, knobs, microphones, personal mobile devices (portable devices, personal phones, personal tablets, the like, and others), and, in some embodiments, in-vehicle devices such as tablets, projectors, virtual reality devices, which are installed in the vehicle or easily removable from the vehicle; and - Interior sensors that provide data about features inside the vehicle, such as vehicle interior temperature, seat weight sensors, and motion detection sensors.

[0104] The view also shows exemplary vehicle outputs 70 and user facilities 34 that can be positioned in the vehicle 10. The outputs 70 include, but are not limited to: - Audio output component, such as vehicle speakers 701; - visual output components, such as vehicle screens 704; - Vehicle kinematic or dynamic actuators, for example those that influence autonomous driving (e.g. vehicle brake, throttle valve or steering gear) - a steering wheel is shown as an example; - Local devices 34 and remote systems 34 / 50 to which the system can provide a variety of information, such as user identification data, user biometric data, user history data, contextual data (weather, road conditions, etc.), instructions or data for use in providing notifications, alerts or messages to the user or to relevant entities, such as authorities, first responders, parents, vehicle dealers (e.g., service department), an operator or owner of an object vehicle 10 or a customer service center, such as the OnStar® Control Center; and - Control components, if not included in these devices, for controlling one of these components.

[0105] The units, subunits and their functions are described in more detail below. V. System Functions and Algorithms - Figure 4V.A. Introduction to Algorithms

[0106] Fig. Figure 4 schematically shows a view of 400 of the system components of Fig. 3 and various interactions between these during the execution of their functions. Functions and interactions are executed according to one or more exemplary algorithms, processes, or routines, which are schematically represented by the sequence shown according to the embodiments of the present technology. For the sake of simplicity, the algorithms, processes, and routines are hereby collectively referred to as processes or procedures.

[0107] Although a single sequence is shown for simplification, any functions or operations in one or more processes, routines, or subroutines of one or more algorithms can be performed by one or more devices or systems. The steps, operations, or functions of the processes are not necessarily shown in a specific order, and the performance of some or all operations is possible and intended to occur in a different sequence. The processes can also be combined or overlapping, for example, by performing one or more operations of one or more processes within one of the other processes. The operations have been explained in the sequence shown for the sake of simpler description and illustration. Operations can be added, omitted, and / or performed simultaneously without deviating from the scope of the appended claims.It should also be noted that the illustrated processes can be terminated at any time.

[0108] In certain embodiments, some or all operations of the processes and / or substantially equivalent operations are performed by a computer processor, such as the hardware-based processing unit 106, a processing unit of a mobile user device and / or the unit of a remote device, which executes computer-executable instructions stored on a non-volatile, computer-readable storage device of the respective device, such as the data storage device 104 of the vehicle system 20.

[0109] The process can end, or any or multiple operations of the process can be performed again. VB System Components and Functions

[0110] In Fig. 4 The input unit 310 receives input signals from a variety of sources. Input sources include vehicle sensors 60, such as cameras, radar, and LiDAR, to name a few, and local or remote devices 34, 50, such as their data storage components. The inputs from the auxiliary devices 34, 50 are received via the vehicle communication subsystem 30. The input sources also include all other units 320, 340, 350 in various embodiments – some of these feedback connections are schematically indicated by an arrow in Fig. 4 shown.

[0111] The input unit 310 consists of the visual clutter sub-unit 312, the driver context sub-unit 314 and the driving context sub-unit 316.

[0112] The visual clutter subunit 312 receives and / or generates data relevant for determining visual clutter in the driving environment in various embodiments. This data includes object data indicating the position of objects in the driving environment as detected by the vehicle's sensors. In the intended embodiments, some or all of this object data originates from one or more other vehicles, mobile devices, or road infrastructure, or is otherwise received by an auxiliary vehicle device. For example, the data may include information about the location and dynamics of other objects—data from a nearby vehicle may, for instance, display position and motion information about that nearby vehicle or another nearby vehicle.

[0113] Motion information can be, for example, an object's trajectory or a predicted position at a specific future time - e.g., in 0.5 seconds from now.

[0114] In various embodiments, the sensor input thus includes two types: the presence of objects and the movement trajectory of objects. At least the presence of objects affects the disorder calculation.

[0115] The system is configured in various embodiments to classify a nearby object as problematic, high-risk, or highly dangerous if its position and dynamics relative to the same object are within a certain threshold for the vehicle 10 in question. For example, if the time-to-collision (TTC), based on current positions and dynamics, is 2 seconds, the object may be classified as high-risk, again depending on the system settings. The threshold can be different, for example, a higher or lower TTC, or the use of other variables, such as a strict distance requirement from vehicle 10.

[0116] The collected data relevant for determining visual clutter are forwarded for use via the Scene Perception Estimation Output Unit 334, which is described in more detail below.

[0117] In various embodiments, the visual clutter data includes a standard object list generated based on data input, such as vehicle sensor data. The standard object list designates one or more objects in the vehicle's environment. As described in more detail below, in some embodiments the system is configured to adapt the standard object list based on additional context—driver context and / or driving context—for example, to obtain a customized object list.

[0118] The driving context sub-unit 316 receives or generates data about the current driving scenario. Examples of driving context information include the road type – shape, condition, etc. – traffic, weather conditions, presence of vehicles or objects in the vicinity, their positions, movement characteristics, vehicle occupancy, vehicle type, or capabilities – such as braking or cornering behavior, and the like.

[0119] The driver context subunit 314 receives or generates data about the conditions affecting the driver, regardless of whether the driver is a human user and / or the vehicle, as in semi-autonomous driving. Driver context information can include information based on, or derived from, the outputs of the other two input modules 312 and 316, as shown in Fig. Figure 4 schematically illustrates how, for example, vehicle occupancy information, weather information, and information about visual disturbances are generated. In various embodiments, driver context information can include any type of information, such as vehicle occupancy information, regardless of whether it was received from another unit 312, 316.

[0120] Examples of driver context data include information indicating the driver's or user's condition, or information about conditions affecting the driver's or user's attention to vehicle operation. Examples of user condition information include visual acuity, physiological measurements, current activities, age, attentional capacity, visual impairments, and other stimuli perceived by the driver.

[0121] Examples of physiological measurements that may affect the driver's attention to the driving scenario include user temperature, electrocardiogram (ECG), electrodermal activity (EDA) or galvanic skin response (GSR), blood volume pulse (BVP), heart rate (HR), electroencephalogram (EEG), electromyography (EMG), blood alcohol or other substance levels, eye blinking or closure or any other visual cue, salinity, similar, other, and any semantic or other interpretation of such a factor. These factors may indicate a certainty or a high probability that the driver or user is anxious, intoxicated, distracted, or otherwise incapable of assessing the necessary provisions regarding the current driving situation, such as how to maneuver or instruct the vehicle.

[0122] As mentioned, in various configurations, the driver's vision is a factor. The system includes a structure for detecting and analyzing the user's vision, such as a semantic vision observer and suitable underlying software. Gaze information might, for example, indicate that the driver is seemingly briefly looking at a child in the back seat, either directly or via a rearview mirror.

[0123] In some cases, the state or condition is generated, at least in part, based on the actions or interactions of multiple users over time. For such user models, the executing system can use this historical multi-user data in combination with current and / or past actions or interactions with a particular user. The historical data can, for example, serve as a basis for adjusting or interpreting current and / or past actions or interactions with the respective user. Model generation may involve considering the user's personal characteristics, such as demographic data (age, gender, height, etc.), and / or characteristics of other users. The system can correlate common characteristics when deriving insights about current actions or interactions with the driver in question.Historical information indicating how attentive or capable a user is of responding to a situation or to stimuli, other users, and / or the user in question can be used to interpret a current, similar situation. Such information can be more useful if commonalities between past and current situations are first established. In one embodiment, the system restricts historical data to data that meets a threshold of significance, for example, by reference to the user in question and / or to a user whose situation is sufficiently similar, as defined by a system designer. User characteristics are explicitly inputted by users in various embodiments and, in some cases, can be learned from the system in the context of experience.The historical data in various forms indicates patterns, user actions, or interactions that can be identified by the system and used to interpret a current situation.

[0124] The driver context sub-unit 314 can use the information in various ways, e.g., to determine whether a user might perceive or fully assess an object or presented vehicle communication.

[0125] In various implementations, the user model is used by the vehicle system during the journey, and the system can also adapt the user model during the journey. The user model can be updated by the system during and / or between journeys with values ​​or other data indicating the driver's activities or tendencies (e.g., reaction time to specific stimuli, such as the time to perceive an object entering the vehicle's path from the right), mannerisms, interactions with the vehicle, preferences, and the like. Updates can also include adding another user, such as a family member, a new user of a rental car, a new fleet vehicle employee, or similar.

[0126] Subunit 314 is configured to generate a perceptual map based on eye-tracking information in various implementations. This generation can occur dynamically, modifying the map in response to other system activities, such as recognizing that a particular object is now in close proximity and / or qualifies as a driving hazard, or that the driver appears to be unaware of or unaware of the hazard.

[0127] The system is configured in various configurations to determine driver or user status based on a dynamic user model. The model can utilize data about the user and / or other users, whether online, in real-world driving situations, and / or offline, such as through queries to the user. The resulting model or user profile can be stored locally, on the vehicle, or remotely, for example, on a cloud server. The user model, which is dynamic in various configurations, incorporates system learning over time to improve the model, for example, based on feedback regarding current and / or historical activities, interactions, etc., for the driver in question and / or other drivers, relating to a single current or previous driver and / or multiple drivers.

[0128] An example of driver context data can also include or be generated from past or present information about a specific driver and / or other drivers. The data might include, for example, statistical information about past performance or experiences with the driver in question or other drivers, such as responses to specific stimuli, reaction times, and the manner in which they react to specific environmental conditions and / or vehicle warnings.

[0129] Supporting data can be received from various sources, including vehicle sensors and sources outside the vehicle, e.g., through crowdsourcing, nearby vehicles, mobile devices, infrastructure, or remote servers.

[0130] Activity unit 320 determines, with continued reference to the structure of Fig.4 the activity results that can be used by the vehicle 10 to determine how to achieve better interaction with the driver, improvement of autonomous driving behavior and / or communication with an additional vehicle device to improve a current driving scenario.

[0131] Improved user interactions may include, for example, earlier notification of the user about a nearby object, a required maneuver (turn, etc.), or other information, or selectively accentuating the notification.

[0132] An example of accentuating notifications is increasing the visibility or volume, sound, number of modalities - from visual alarm to visual and haptic alarm and audible alarm - when providing an alert - or other methods to increase the likelihood that the user will notice the notification.

[0133] Accentuation in various forms involves highlighting a particular component of a notification, e.g., by enlarging, brightening, or using a different color on a screen; an important or dangerous object that the user may not fully perceive or be able to assess—such as a notification that merely indicates the presence of the object, or a time or distance until impact.

[0134] In the intended embodiments, notification is provided via a heads-up display or a type of virtual reality display, positioned at the vehicle's location so that the driver sees the notification in conjunction with their perception of the actual object or area. For example, if the system determines that, due to driver distraction or high levels of disturbance, the driver is unable or unlikely to assess the immediate proximity of a pedestrian or the need to make a turn for navigation or to avoid an object, the system can highlight the object or area in a display next to or on the vehicle's windshield or window.

[0135] In various embodiments, a notification for a vehicle operator is made available to the user via a user device, such as a telephone or smartwatch. This arrangement can be particularly useful in partially or fully autonomous driving situations where the user may not be in a position or otherwise prepared to perceive notifications intended to be received by a driver in a driving position and attitude.

[0136] Instead of or with such highlighting, the system can provide relevant text, symbols, audio (sound, words, etc.), haptic output, etc., through any vehicle output components. While in some embodiments the highlighting is provided at a basic level in response to system determinations, in other embodiments the highlighting of the vehicle 10 is enhanced in response to the determinations, such that a lower notification level is provided when the driver or user is not detected as distracted or otherwise less likely to fully perceive the driving scene.

[0137] With regard to partially or fully autonomous driving, the results of Activity Unit 320's activity may include accentuated instructions that cause the vehicle to perform maneuvers more safely or otherwise more effectively. For example, if Unit 320 determines that the driving environment is very obscured, as in the aforementioned Times Square example, and especially in heavy rain, glare, or other factors that restrict visibility, the activity results may cause the vehicle to slow down compared to when considering only the distance to a leading vehicle.In the case of adapting an autonomous driving function in a scenario with high obscurity, one reason for such a change is to compensate for the possibility that determinations of presence and position – from vehicle sensors, from other vehicle data received from other vehicles or from nearby infrastructure, etc. – may not be as reliable as they would be in a scenario with low obscurity.

[0138] In various embodiments, the output structuring unit 330 includes an alarm evaluation output sub-unit 332 and the scene perception output unit 334 to process the output of the user-centric alarm evaluation sub-unit 322 or the scene perception output unit 334.

[0139] Output subunits 332 and 334 can each, or together, include an automated decision-making system or be referred to as a subsystem. The subunits determine instructions, messages, signals, or other suitable structures based on the point and clutter information of the environment, which is output by the user-oriented alarm evaluation subunit 322 and the scene perception estimation output unit 334.

[0140] In various embodiments, the user-oriented alarm assessment subunit 322 determines a hazard assessment value or score, or other useful output, based on the aforementioned driver context information. The user-oriented alarm assessment subunit 322 can also consider the aforementioned driving context information and environmental information separately, or, as indicated, incorporate one or both of these pieces of information into the driver context information.

[0141] The user-oriented point value for alarm assessment can be created according to any suitable scale, protocol or format, such as on a scale of 0.0 to 10.0, 1 to 1000, level 1-5 or similar.

[0142] The user-oriented alarm assessment value can be used by the vehicle 10 or other devices, such as nearby vehicles or user devices - pedestrian telephones, etc. - to determine the next actions.

[0143] Further actions may include, for example, controlling the vehicle, initiating communication with the user, and initiating control of the operation or communication to (and possibly to and from) the other device.

[0144] In various embodiments, the Scene Assessment 324 uses sensor data and / or other data, such as from an additional vehicle device, to calculate a level of visual clutter in an environment. This clutter can be based, for example, on camera and radar inputs, and may include object trajectory data and data indicating highly hazardous objects.

[0145] The degree of disorder is used in various implementations to estimate the probability of the driver perceiving one or more specific objects in the driving environment. With a higher degree of disorder, the probability of the driver perceiving an object is generally lower. This probability is used to determine an alarm strategy that informs the driver about the object(s).

[0146] As with user-oriented hazard assessment, the level of disorder from vehicle 10 or other devices, such as nearby vehicles or mobile communication devices, can be used to determine the next actions. Again, the next actions may include, for example, controlling the vehicle, initiating communication with the user, and initiating control of the operation or communication to (and possibly to and from) the other device.

[0147] In various embodiments, the user-oriented hazard assessment is determined using at least one of two techniques: a rule-based technique and a machine-based technique.

[0148] The rule-based technique obtains or determines a collision avoidance time (ACT). Although the term focuses on collisions, typically referring to those involving a vehicle body, the concept can be applied to other objects, such as potholes, or maneuvers, like a vehicle turn required to follow navigation. The ACT is determined based on any given driver context, driving context, and clutter information.

[0149] ACT represents the time a driver needs to react to a hazard or to execute a necessary maneuver. A driver who shows no impairments (e.g., vision, attention) or distractions (e.g., phone call, disorganization) can be considered a standard driver and assigned a standard ACT, for example.

[0150] A driver's reaction time is lower when arguing with a passenger than when driving alone and relaxed. When the driver's assessed capacity is lower, the corresponding ACT (Acute Action Task) is higher in relation to the driver and any nearby objects. One possible way to address this capability deficit is to inform the driver earlier about an approaching object, an object being approached, a required maneuver, or similar situations.

[0151] In various implementations, ACT is determined as a function of Time-to-Collision (TTC) – the time elapsed before an event, such as a collision, hitting a pothole, or reaching a point in the required maneuver. TTC is based on the current dynamics (vehicle and object distance and dynamics) and is not user-specific.

[0152] In various embodiments, the system is configured to adapt ATC to a new, customized ACT for each context and each driver, depending on one or more factors such as driver context, driving context, clutter information, and TTC.

[0153] As an example, let's assume that the ACT corresponding to an older driver talking on the phone is 3 seconds. Therefore, the driver should be warned at an ACT of at least 3 seconds, even if the system is normally only activated after a TTC of 1.8 seconds.

[0154] For the rule-based technique for determining the hazard assessment value, the alarm assessment output unit 332 includes a code that defines at least one rule for determining the value based on various factors.

[0155] For example, a rule might state the following: If (driver on the phone) and (age > 55 years), then ACT = 3 sec.

[0156] As an example, another rule might state the following: If (weather = rain) and (age > 55), then ACT = 3 sec.

[0157] If the specified ACT is greater than the TTC, then perform action X, in which: X = warns the driver of the event (collision, pothole, etc.) earlier than the TTC time and at least until the ACT time before the event.

[0158] If the specified ACT is less than the TTC, then Y, where: Y = The warning is sufficient for TTC and acceptable for ACT.

[0159] In some embodiments, Y also includes advising the user, such as suggesting that the user advise a child who has stepped into the road and was not easily or early spotted to proceed more cautiously, or advising the user to be more attentive or less distracted (e.g., not to use a mobile phone while driving).

[0160] For the machine learning approach to determining the hazard assessment score, the alarm output unit 332 is configured with code to statistically determine ACT, e.g., based on a probability function. In various embodiments, the probability function is learned over time based on user performance in similar or different situations.

[0161] In some implementations, the warning assessment output unit 332 is configured so that under certain circumstances (which may include, among others, the prevailing driver context, the driving context, and clutter information) a current situation or object may be classified as problematic or dangerous, even if a given ACT value is less than or earlier than a TTC for the object.

[0162] In various implementations, machine learning uses deep learning or a theorem such as Bayes' theorem to determine the probability that a current state is problematic. Processing with respect to the probability (P) that the current state (CS) is dangerous might, for example, involve the following: ▪ If TTC > threshold (T), then P(CS) = dangerous (i.e., the current state is likely dangerous) and should therefore perform action X, where X includes the following: ▪ Adapting the TTC to the ACT; and / or ▪ Initiating further actions determined by the automated decision support components linked to the alert assessment output unit 332, such as determining an action based on the driver context, the driving context and / or the clutter information - e.g., highlighting a notification to the user, sending a message to a mobile pedestrian device, assuming some driver control for temporary autonomous handling under these circumstances. ▪ If TTC < threshold (T), then P(CS) ≠ dangerous, and therefore no separate intervention is required.

[0163] The threshold (T) is a confidence level in various implementations, and the Scene Perception Estimation output subunit 334 determines that a current state is problematic if the TTC is above the threshold (T).

[0164] As previously mentioned, the user-oriented hazard assessment value can be used by vehicle 10 or other devices, such as nearby vehicles or user devices (e.g., pedestrian telephones), to determine the next actions. These next actions may include, for example, controlling vehicle 10, communicating with the user of vehicle 10, operating the vehicle, communicating with the respective users of the other device, or controlling the other devices.

[0165] The instructions or controls are configured to increase the likelihood that the user will perceive one or more objects nearby or will need to perform another maneuver, such as during the following navigation.

[0166] In addition to the embodiments where environmental clutter is considered when determining user context, environmental clutter can also be used in the system's scene perception functions. The functions performed by the Scene Perception Estimation Output Subunit 334 in various embodiments include, among other things, customizing the aforementioned standard object list, resulting in the customized object list. The standard object list displays, for example, for each relevant object, an arbitrary position of the object, the direction of motion of the object, the rotational speed or velocity of the object, the size or other dimensions or geometry of the object, and the object type.

[0167] The adaptation is based on various factors, such as the driver context and / or the driving context. The adapted object list can be created to take into account factors that might distract the driver to some extent from appreciating the objects while driving and reacting to them fully and appropriately, or that might prevent the driver from acknowledging or reacting appropriately to the vehicle's warnings regarding driving, such as in the case of fully human driving or semi- or fully autonomous driving.

[0168] As a simple example, the adapted object list can shift the effective position of an object, such as a high-risk object like a pedestrian, to a position closer to the vehicle, thus informing the user earlier of the object's presence or proximity. This can be useful if the system has information or has determined that the user has slow reflexes or is otherwise impaired and could benefit from early notification.

[0169] In addition to the functions mentioned in the standard object list, the adapted object list can include a perception or probability factor [P(perception) or simply (P)] regarding a nearby perceived object, or be used to indicate a probability that the driver perceives or fully assesses the presence, dynamics, or danger—for example, the immediate proximity—of a nearby object.

[0170] Variables used to generate the probability of perception include object information (position, dynamics, etc.), user's gaze (e.g., direction, time, or timing), visual clutter, driver attention, or other indicators of the driver context.

[0171] The probability (P) can be expressed according to any suitable scale, format or protocol, such as a percentage of probability (0-100%), a scale of 1-10, the like, or others.

[0172] In various embodiments, the probability (P) that the user perceives a particular object (e.g., a nearby vehicle) is set to or maintained at a preset value, such as 1, when the basic requirements are met. Examples of requirements include: (i) the driver's gaze is directed toward the object in question (e.g., at some point in the last two seconds or any time period a designer deems appropriate for using the system), (ii) the driver is not drowsy, and (iii) the visual clutter in the scene is below a critical threshold (T). Otherwise, the probability is set to 2.

[0173] Or the base level can be 0, which can be set to P0 if the requirements are met, and 1 otherwise. Or any effective convention.

[0174] These approaches can be described as the rule-based technique for determining a probability (P) with which the user perceives an object.

[0175] In some embodiments, the system is configured to calculate the probability (P) using a learned regression and / or simulation environment. The probability (P) can be determined, for example, as a learning regression of input parameters, taking into account all data collected in a simulation environment. This approach can be referred to as machine learning.

[0176] As with the user-oriented hazard assessment value, instructions or controls are generated that are implemented on the vehicle 10 or on other devices - other vehicles, pedestrians, etc. - and can be configured based on the adapted object list or perception probability to increase the likelihood that the user will perceive one or more objects nearby or that they will need to perform a maneuver to follow the navigation or avoid a hazard.

[0177] Instructions for influencing vehicle control, the control of other devices, or communication with the users of the vehicle or other devices are configured in various embodiments, based on one or more of the user-oriented hazard assessment points, the adapted object list, and the probability of perception (P).

[0178] The implementation unit 340 includes, as intended, an output interface sub-unit 342. The output interface sub-unit 342 processes the output of the activity unit 320 in any way possible to send resulting instructions, messages, or signals to destinations. Exemplary destinations include a vehicle system, such as an in-vehicle communication system or an autonomous driving system. Other destinations include, for example, receiver devices for auxiliary vehicles, to which instructions are sent for communicating with a user of the device or for controlling the device. The processing can also include formatting the output, for example, for use at the destination.In various embodiments, the processing includes preparing the signal or messages for transmission, which may also include initiating the transmission to the destination and actually carrying out the transmission.

[0179] The implementation unit 340 or a target device - vehicle communication or autonomous driving systems, other vehicle systems, etc. - performs communication via the vehicle or device, vehicle or device control, or other action. VI. Additional Structure, Algorithm Features and Operations

[0180] In combination with or instead of any of the other embodiments described herein, the present technology may include any structure or perform any functions as follows: a) In various embodiments, the existing systems and processes quantify an alert or hazard level by integrating driver context, driving context and / or clutter information. b) In various embodiments, user-oriented alarm assessment and scene perception functions output values ​​and / or structures - such as an alarm assessment result and a customized object list - that capture more precise information about a driving situation, including the urgency of the driving situation, such as time until collision, potential damage, other vehicles or persons involved, such as pedestrians, drivers or passengers. c) The results of the alert assessment can be used to determine whether measures are needed to reduce the urgency level. Examples of actions include: Providing prior notification to the driver to ensure that they perceive the object and react accordingly or perform any necessary maneuvers; Providing a decorated or highlighted notification – such as one that is emphasized, with a higher volume, certain colors, the like, or similar features – to the driver to ensure that he or she recognizes the object and reacts appropriately to it or performs a necessary maneuver accordingly; Determination and implementation of a staged warning system to avoid false alarms if the TTC is set earlier than usual; Sending notifications to auxiliary vehicle devices, such as other vehicles and mobile pedestrian devices; Displaying warnings or notifications in the vehicle by modalities that are most appropriate under the given circumstances - e.g., by haptic warning, with or without other modalities, when the noise level in the vehicle cabin is high; Controlling or adjusting vehicle systems, such as settings for automatic cruise control, other autonomous driving functions, infotainment settings (e.g. radio volume), the like, and others; Transferring partial or full driving responsibility to the driver from any level of autonomous driving – the transfer takes place in various forms with or after notifying the driver that the transfer is taking place – e.g., “User intervention required; please take the steering wheel” or other advisory and / or reassuring communication; and Transferring partial or full driving responsibility from the driver to the vehicle - the system can accompany the transfer with a notification that the transfer is taking place and / or that everything is OK - e.g., "Vehicle intervening to temporarily assist with driving", "All under control", or other advisory and / or reassuring communication. d) Each of the functions described herein can be performed using artificial intelligence (AI) techniques. e) In various embodiments, the current systems and processes for supporting the determination of how to adapt vehicle operations or initiate communication with the vehicle or other devices – other vehicles, pedestrians, etc. – perform any of the following functions: (1) detecting the vehicle's driving scene (road conditions, driving in the oncoming lane, hazardous situation, crossing, pedestrians, obstacles), (2) determining a driver state, and (3) determining a driving context and interpreting these inputs together to decide on a score associated with the level of hazard of the situation. f) In various embodiments, the technology effectively filters out environmental noise or other inputs to the driver related to driving, making driving easier and highlighting important information that the driver would otherwise not perceive or understand. Examples include vehicle instructions or objects in the driver's field of view (FOV). (g) In various embodiments, the system enables a process for estimating the driver's perception of one or more objects near the vehicle. The system is configured to estimate the probability that the driver perceives a particular object in the scene. The system can use this measure to enhance or otherwise influence the alarm strategy of an active safety feature—for example, a driver assistance system. In some implementations, this results in a lower overall number of warnings while maintaining a level of safety equivalent to what the vehicle would provide without the enhancement. VII. Selection of advantages

[0181] Many of the advantages and benefits of this technology have been described above. This section highlights some of them again and also refers to others. The advantages described are not exhaustive.

[0182] The technology increases user acceptance and trust in the vehicle and its active safety features.

[0183] The system reduces driver distraction by providing the driver with relevant information according to current conditions – such as driving context, driver context and clutter – and / or by providing the information earlier and / or in a highlighted manner, avoiding hazards earlier and indicating maneuvers earlier, thereby avoiding the need for advanced, higher-level warning messages.

[0184] The system, in its various embodiments, operates as a semantic parser of inputs captured by the vehicle, whereby the resulting vehicle notifications to the user and, if necessary, the vehicle control are tailored to the current situation – including driver context, driving context and / or clutter information – and not, as with conventional systems, based solely on the relative position of the vehicle and nearby objects. VIII. Conclusion

[0185] Various embodiments of the present disclosure have been disclosed herein. The disclosed embodiments serve only as examples, which can be developed in various alternative forms and combinations thereof.

[0186] The embodiments described above are merely illustrative examples of implementations to facilitate an easy understanding of the basic ideas of the disclosure.

[0187] References herein concerning how a feature is arranged may relate to, among other things, how the feature is positioned relative to other features. References herein concerning how a feature is configured may relate to, among other things, the feature's size, shape, and / or material. For simplicity, the term "configured" may be used to refer to both the configuration and device described above in this paragraph.

[0188] Orientable references are provided here primarily to simplify the description and the illustrative drawings, and the described systems can be implemented in any of a wide variety of orientations. References herein indicating direction are not made in boundary sensors. For example, references to top, bottom, above, below, or side are not provided in order to limit the way in which the technology of this disclosure can be implemented. While, for example, an upper surface is referenced, the referenced surface may not be vertically upward or above in a design, manufacturing, or operating reference frame. The surface may, for example, be to the side or below other components of the system in various embodiments.

[0189] Each component described or shown in the figures as a single element can be replaced by several such elements configured to perform the functions of the single item. Likewise, any number of elements can be replaced by a single element configured to perform the functions of the various described elements.

[0190] The embodiments described above may be modified, altered, or combined without altering the scope of protection of the claims. These modifications, alterations, and combinations are included within the scope of protection of this disclosure by the following claims.

Claims

[1] User-oriented driver assistance system, including: at least one vehicle sensor (60, 134); a hardware-based processing unit (106); and a non-volatile computer-readable storage device (104), comprising: an activity unit (320) which, when executed by the hardware-based processing unit (106), determines, based on context-dependent input information, at least one of an alarm assessment output and a scene perception output, wherein the context-dependent input information includes or is generated by the output of the vehicle sensor (60, 134); and an output structuring unit (330) which, when executed by the hardware-based processing unit (106), determines an action to be performed on a vehicle (10) based on at least one of the alarm assessment outputs and the scene perception output determined by the activity unit (320), wherein the activity unit (320), when executed by the hardware-based processing unit (106): receives or generates a standard object list that was created at least on the basis of the output of the vehicle sensor (60, 134); and adapts the standard object list based on the context-dependent input information and outputs a customized object list; and The scene perception output includes the adapted object list. [2] User-oriented driver assistance system according to claim 1, wherein the context-dependent input information is based at least partially on a user model generated using past features of a current user and / or other users. [3] User-oriented driving assistance system according to claim 1, wherein the context-dependent input information includes a user context relating to a state or condition of a user of the vehicle (10). [4] User-oriented driver assistance system according to claim 3, wherein the user's state or condition is determined based on (i) biometric data from user measurements, (ii) user activity, (iii) vehicle occupancy and / or (iv) user gaze. [5] User-oriented driving assistance system according to claim 1, wherein the context-dependent input information includes a user context relating to the current driving conditions. [6] User-oriented driving assistance system according to claim 1, wherein the context-dependent input information includes clutter information relating to a quantity or quality of visual objects in an environment outside the vehicle (10). [7] User-oriented driving assistance system according to claim 1, wherein the alarm assessment output includes an alarm assessment value. [8] User-oriented driver assistance system according to claim 1, wherein the action determined by the output structuring unit (330) comprises controlling one or more of the following steps: (i) a time at which a vehicle-related notification is transmitted via a vehicle output device, (ii) a substance of the vehicle-related notification, (iii) a modality or modalities for use in providing the notification, (iv) a highlighting for use in providing the notification, and (v) a type of highlighting for use in providing the notification. [9] User-oriented driver assistance system according to claim 1, wherein the action determined by the output structuring unit (330) comprises one or both of the following components: controlling an automated drive component of the vehicle (10); and generating communication for transfer to a target device other than the vehicle (10).

Images