Method and user information system for increasing a user's trust in an automated vehicle

Abstract

User information system (111) for a motor vehicle (110) with an automated driving mode, in which the vehicle (110) is automatically guided longitudinally and transversely; wherein the user information system (111) is set up while the vehicle (110) is operated in the automated driving mode, - to determine environmental data relating to the environment of the vehicle (110), - by means of an estimation unit of the user information system (111), on the basis of the environment data, to estimate a sub-area of the environment observed by an assumed driver (130) manually driving the vehicle (110); and - outputting environmental information relating to the estimated sub-area of the environment to a user of the vehicle (110).

Description

[0001] The invention relates to an automated vehicle. In particular, the invention relates to a method and a corresponding user information system designed to increase a user's confidence in the automated driving operation of an automated vehicle.

[0002] A vehicle may have a driving system configured to automatically guide the vehicle longitudinally and laterally. The automated longitudinal and lateral guidance may be implemented in such a way that the vehicle user can divert their attention from the driving task, at least temporarily. The effectiveness with which the user can use their time in the vehicle for other tasks depends in particular on the user's confidence that the vehicle will reliably perform the driving task automatically.

[0003] From the document DE 10 2014 214 506 A1, a method for creating an environment model of a vehicle is known, in which sensor data from a sensor system is received, wherein at least one object is determined based on the sensor data, wherein electromagnetic radiation received by the sensor system from at least one object is visually displayed to an observer by means of a display device, wherein an object class from a predefined list of object classes is assigned to the object, and wherein the assignment of the object to an object class is displayed to the observer. This can strengthen the trust of drivers (and passengers) in driver assistance systems that enable a higher degree of automation. The observer is able to understand how the driver assistance system reacts to objects in the vehicle's environment.In particular, the viewer can see which objects are included in the environment model and which are not.

[0004] The document DE 10 2015 212 664 A1 describes a motor vehicle with an automatic driving system which comprises a sensor for detecting the surroundings of the motor vehicle and an object selection module for determining at least one control object, wherein the motor vehicle can be driven automatically by means of the automatic driving system depending on the position and / or the speed of the control object, wherein the motor vehicle comprises a viewing direction sensor for detecting the viewing direction of the driver of the motor vehicle and a windshield display for displaying variable information on the windshield of the motor vehicle, wherein the motor vehicle comprises a transformation module for aligning a marking displayed / displayable by means of the windshield display in such a way that the marking marks the control object from the perspective of the driver of the motor vehicle.

[0005] Furthermore, DE 10 2015 220 249 A1 discloses a method for the automated driving of a motor vehicle using a driver assistance system, wherein driving-relevant information from the driver assistance system is displayed to the driver on a visual display device. To provide the driver with simple and clear information, it is proposed that the driving-relevant information be displayed on the display device using a limited number of different display pages assigned to different information classes, with the display pages being displayed sequentially and in a fixed order. This method can significantly increase the driver's confidence in the driver assistance system.For example, the driver can obtain an overview of surrounding objects detected by the vehicle's environment detection system in order to ensure that the vehicle control system and thus also the automated driving of the vehicle are working reliably.

[0006] Reference is also made to publication DE 10 2016 201 939 A1. If a driver's sudden increase in attention is detected by suitable sensors, the driver's line of sight can also be recorded in such a situation. This means that a hazard detected by the driver, including the driver's line of sight, can be detected by sensors. The information detected by the driver is then taken into account when determining the hazardous situation. This determination includes identifying objects in the vehicle's vicinity and their position, as well as determining whether these objects are sufficiently certain to exist. Interventions in the vehicle's driving behavior or warning measures for the driver are only justified if the existence of the objects is sufficiently certain.

[0007] This document addresses the technical task of increasing the confidence of a vehicle user in the automated driving operation of the vehicle, in particular in order to increase the comfort and benefit of the automated driving operation of the vehicle for the user.

[0008] The problem is solved in each case by the independent claims. Advantageous embodiments are described, among other things, in the dependent claims. It should be noted that additional features of a patent claim dependent on an independent patent claim can form a separate invention, independent of the combination of all features of the independent patent claim, without the features of the independent patent claim or only in combination with a subset of the features of the independent patent claim, which invention can be made the subject of an independent claim, a divisional application, or a subsequent application. This applies equally to technical teachings described in the description, which can form an invention independent of the features of the independent patent claims.

[0009] According to one aspect, a user information system for a motor vehicle with an automated driving mode is described. In the automated driving mode, the vehicle is automatically guided longitudinally and / or laterally. In particular, the vehicle can be guided longitudinally and laterally in the automated driving mode according to SAE Level 3. The responsibility for driving operation can thus lie with the vehicle, and the user of the vehicle (in particular the user in the driver position of the vehicle) can, if necessary, have the freedom to deal with matters other than the driving task and / or monitoring the driving task during operation of the automated driving mode.

[0010] The user information system is configured to determine environmental data relating to the vehicle's surroundings while the vehicle is operating in automated driving mode. The environmental data may include sensor data from one or more environmental sensors, in particular from at least one camera, at least one lidar sensor, and / or at least one radar sensor, of the vehicle. The environmental data may describe a current traffic and / or driving situation of the vehicle. In particular, the environmental data may indicate one or more objects (in particular one or more other road users) in the vehicle's surroundings.

[0011] The user information system can be configured to use an estimation unit, based on the environmental data, to predict or estimate a sub-area of the environment that a driver of the vehicle would likely view if the driver were driving the vehicle manually. In other words, a (pre-trained) estimation unit can be used to determine which sub-area of the environment a driver of the vehicle would view if the driver were driving the vehicle themselves. In still other words, based on the environmental data, a sub-area of the environment can be predicted or estimated that an assumed driver driving the vehicle manually would view (probably, or with a probability greater than a certain probability threshold, such as 30% or 50%).This makes it possible to determine the part of the environment that has the relatively highest probability of being viewed by a driver (compared to other parts of the environment).

[0012] The estimation unit can be trained, for example, using a machine learning method. Alternatively or additionally, the estimation unit can be trained using training data that, for a variety of different driving and / or traffic situations, indicate the part of the environment that was actually observed by the respective driver. The estimation unit can optionally include at least one neural network trained (based on the training data).

[0013] In particular, the estimation unit may have been trained in such a way that the estimation unit is designed to take into account a typical gaze behavior of one or more drivers when manually driving a vehicle in order to predict or estimate the part of the environment.

[0014] Thus, a previously trained estimation unit can be used to predict or estimate, on the basis of the current environmental data, the part of the vehicle's environment that would presumably be relevant for the user of the vehicle in the current driving and / or traffic situation if the user were to drive the vehicle themselves.

[0015] The estimation unit can be trained in such a way that the estimation unit is configured to estimate the correct sub-area of the environment that a driver, assuming the vehicle is manually driving, would observe with a reliability greater than a certain probability threshold. The probability threshold can be determined experimentally in advance. The probability threshold can thus depend on the estimation unit. The estimation unit can, in particular, be configured to indicate the reliability of the estimate for an estimated sub-area of the environment, in particular to indicate the probability with which the estimated sub-area is the correct sub-area (i.e., the one that a driver manually driving the vehicle would actually observe).

[0016] Furthermore, the user information system is configured to output environmental information relating to the estimated sub-area of the surroundings to the vehicle user. The environmental information relating to the estimated sub-area of the surroundings can be output, in particular, via a loudspeaker and / or on a screen of the vehicle. The environmental information can be output in such a way that the user becomes aware that the vehicle is taking the estimated sub-area of the surroundings into account as part of the automated driving mode.

[0017] By predicting or estimating the relevant part of the environment and outputting environmental information related to the estimated part of the environment, the user can be informed that the relevant part of the environment is being taken into account during automated driving of the vehicle. This can increase the user's confidence in the automated driving mode, thus increasing the comfort and benefit of the automated driving mode for the user.

[0018] The user information system can be configured to detect one or more objects (e.g., one or more other road users) in the vicinity of the vehicle based on the environmental data. The estimation unit can then predict or estimate which of the one or more objects a driver would likely view if the driver were driving the vehicle manually, or which of the one or more objects an assumed driver driving the vehicle manually would view (with a relatively high probability). In particular, it can be predicted or estimated which object would likely be viewed by the user (or with a probability greater than the probability threshold) if the user were driving the vehicle manually.

[0019] Environmental information relating to the estimated object can then be provided to the vehicle user, particularly to inform the user that the estimated object is being taken into account within the automated driving mode. By taking one or more specific objects into account, the user's confidence in the automated driving mode can be further increased.

[0020] Alternatively or additionally, the user information system can be set up to determine a traffic situation with one or more other road users in which the vehicle is located on the basis of the environmental data. Furthermore, the estimation unit can be used to predict or estimate which of the one or more other road users a driver would probably look at if the driver were driving the vehicle manually, or which of the one or more other road users an assumed driver driving the vehicle manually would look at (with a relatively high probability, for example with a probability that is higher than the probability threshold). Environmental information relating to the estimated road user can then be output to the user of the vehicle.By estimating one or more road users relevant to the user, the user's confidence in the automated driving mode can be further increased.

[0021] The user information system can be configured to output an image relating to the surroundings of the vehicle, in particular relating to one or more objects (such as road users) in the surroundings, on a screen of the vehicle. The image can include, in particular, a synthetic and / or graphical representation of the surroundings with one or more object symbols for the one or more objects in the surroundings of the vehicle.

[0022] Furthermore, the user information system can be configured to highlight the estimated sub-area of the surroundings, in particular an estimated object in the surroundings, of the vehicle within the image. In particular, the object symbol for the estimated object can be highlighted compared to the one or more object symbols for one or more other objects in the surroundings of the vehicle. The highlighting can be achieved, for example, by adjusting the color, intensity, and / or size of the object or object symbol to be highlighted. By highlighting an object relevant to the user in a visual representation of the current traffic situation, the user's confidence in the automated driving mode can be further increased.

[0023] The user information system can be configured to determine feedback from the user as to whether the estimated area of the surroundings corresponds to the area of the surroundings that was actually viewed by the user and / or was actually relevant to the user. The feedback can be collected, for example, via a user interface of the vehicle.

[0024] The estimation unit can then be adjusted based on the feedback. The feedback, along with the surrounding data, can be used as a training dataset to teach the estimation unit. This allows the estimation unit to be individually adapted to the vehicle user. This can increase the accuracy of the prediction or estimation of relevant parts of the environment, further increasing the user's confidence in the automated driving mode.

[0025] According to a further aspect, a device for training an estimation unit is described. The estimation unit is intended to be capable of estimating, based on environmental data relating to the surroundings of a motor vehicle, a sub-area of the surroundings that a driver of the vehicle is likely to observe if the driver is manually driving the vehicle, or that an assumed driver manually driving the vehicle would observe (with a probability higher than the probability threshold).

[0026] The device is configured to determine a plurality of training data sets. Each training data set includes environmental data relating to the surroundings of a vehicle during a test drive, as well as information relating to the portion of the surroundings of the vehicle that was actually viewed by the driver of the vehicle during the test drive. The device is further configured to train the estimation unit (e.g., a neural network of the estimation unit) based on the plurality of training data sets. This allows training of an estimation unit that reliably predicts or estimates the gaze behavior of motor vehicle drivers.

[0027] According to a further aspect, a (road) motor vehicle (in particular a passenger car or a truck or a bus or a motorcycle) is described which comprises the device described in this document and / or the user information system described in this document.

[0028] According to one aspect, a method is described for increasing the confidence of a user of a motor vehicle in an automated driving mode of the vehicle, wherein the vehicle is automatically guided longitudinally and transversely in the automated driving mode. The method comprises, while the vehicle is operated in the automated driving mode, determining environmental data relating to the surroundings of the vehicle. Furthermore, the method comprises predicting or estimating, by means of an estimation unit and based on the environmental data, a sub-area of the surroundings that a driver of the vehicle would likely observe if the driver were driving the vehicle manually, or which an assumed driver driving the vehicle manually would observe. The method further comprises outputting, in particular displaying, environmental information relating to the estimated sub-area of the surroundings to the user of the vehicle.

[0029] According to a further aspect, a method for training an estimation unit is described in order to enable the estimation unit to estimate, on the basis of environmental data relating to the environment of a motor vehicle, a sub-area of the environment that a driver of the vehicle probably and / or with a relatively high probability observes if the driver is driving the vehicle manually, or that an assumed driver driving the vehicle manually would observe. The method comprises determining a plurality of training data sets. A training data set comprises environmental data relating to the environment of a vehicle during a test drive and information relating to the sub-area of the environment of the vehicle that was actually observed by the driver of the vehicle during the test drive. The method further comprises training the estimation unit based on the plurality of training data sets.

[0030] According to a further aspect, a software (SW) program is described. The SW program can be configured to be executed on a processor (e.g., on a control unit of a vehicle) and thereby to perform at least one of the methods described in this document.

[0031] According to a further aspect, a storage medium is described. The storage medium can comprise a software program configured to be executed on a processor and thereby to carry out at least one of the methods described in this document.

[0032] It should be noted that the methods, devices, and systems described in this document can be used both alone and in combination with other methods, devices, and systems described in this document. Furthermore, any aspects of the methods, devices, and systems described in this document can be combined in a variety of ways. In particular, the features of the claims can be combined in a variety of ways.

[0033] For the purposes of this document, the term "automated driving" can be understood as driving with automated longitudinal or lateral guidance, or autonomous driving with automated longitudinal and lateral guidance. Automated driving can, for example, involve extended driving on the highway or temporary driving during parking or maneuvering. The term "automated driving" encompasses automated driving with any degree of automation. Examples of levels of automation include assisted, partially automated, highly automated, or fully automated driving. These levels of automation were defined by the Federal Highway Research Institute (BASt) (see the BASt publication "Research Compact," issue 11 / 2012). In assisted driving, the driver continuously performs longitudinal or lateral guidance, while the system assumes the other function within certain limits.In partially automated driving (TAF), the system assumes longitudinal and lateral guidance for a certain period of time and / or in specific situations, whereby the driver must continuously monitor the system, as in assisted driving. In highly automated driving (HAF), the system assumes longitudinal and lateral guidance for a certain period of time without the driver having to continuously monitor the system; however, the driver must be able to assume control of the vehicle after a certain time. In fully automated driving (VAF), the system can automatically handle driving in all situations for a specific application; for this application, a driver is no longer required. The four levels of automation mentioned above correspond to SAE Levels 1 to 4 of the SAE J3016 standard (SAE - Society of Automotive Engineering). For example, highly automated driving (HAF) corresponds to Level 3 of the SAE J3016 standard.Furthermore, SAE J3016 also specifies SAE Level 5 as the highest level of automation, which is not included in the BASt definition. SAE Level 5 corresponds to driverless driving, in which the system can automatically handle all situations like a human driver throughout the entire journey; a driver is generally no longer required. The measures described in this document specifically apply to a vehicle operated according to SAE Level 3 or higher.

[0034] The invention will be described in more detail below using exemplary embodiments. Fig. 1a an exemplary traffic situation in the environment of an ego vehicle; Fig. 1b exemplary components of an ego vehicle; Fig. 1c an exemplary viewing direction of a driver in an exemplary environmental situation; Fig. 2a an example neural network; Fig. 2b an example neuron; Fig. 3 an exemplary image relating to an environmental situation of an ego vehicle; Fig. 4a is a flowchart of an exemplary method for training an estimation unit; Fig. 4b a flowchart of an exemplary method for outputting environmental information with respect to an environmental situation.

[0035] As stated at the beginning, this document is concerned with increasing a user's confidence in the reliability of an automated vehicle, i.e., an automated driving mode of a vehicle. In this context, Fig. 1a shows an exemplary traffic situation of an ego vehicle 110 on a multi-lane (actual) roadway 100 with an ego lane 101 on which the ego vehicle 110 is traveling, and with one or more neighboring lanes 102, 103. In the vicinity of the ego vehicle 110 there are several different other road users 120 (in particular other vehicles).

[0036] Fig. 1b shows exemplary components of an ego vehicle 110. The ego vehicle 110 includes one or more environmental sensors 112 configured to capture environmental data (i.e., sensor data) relating to the environment of the ego vehicle 110. Exemplary environmental sensors 112 include an image camera, a radar sensor, a lidar sensor, an ultrasonic sensor, etc.

[0037] Furthermore, the ego vehicle 110 can include one or more actuators 113 for automated longitudinal and / or lateral guidance of the ego vehicle 110. Examples of actuators 113 are a drive motor, a braking device, and / or a steering device.

[0038] In addition, the ego vehicle 110 can include a screen 114 that is configured to display images with N x M pixels, where N and / or M can be 100 or more, 200 or more, or 500 or more. The screen 114 can, for example, include a TFT (thin-film transistor) screen and / or an LCD (liquid crystal display) screen and / or an LED (light-emitting diode) screen. Alternatively or additionally, the screen can include a head-up display and / or a projector configured to project an image onto a surface. The screen 114 can be arranged in the interior of the ego vehicle 110 and / or on a dashboard or on a center console of the ego vehicle 110.

[0039] A control unit 111 of the ego vehicle 110 can be configured to determine an environment model with respect to the environment of the ego vehicle 110 based on the environment data. The environment model can, for example, describe one or more other road users 120 (generally one or more objects) in the environment of the ego vehicle 110 and / or the number of lanes 101, 102, 103 of the roadway 100 traveled by the ego vehicle 110. The control unit 111 can be configured to operate the one or more actuators 113 of the ego vehicle 110 depending on the environment model, e.g., to provide a driver assistance function (such as a lane keeping assistant, a lane change assistant, adaptive cruise control, etc.) and / or an automated driving mode (e.g., according to SAE Level 3 or higher).

[0040] Furthermore, the control unit 111 can be configured to determine an image relating to the surroundings of the ego vehicle 110 based on the surroundings data and / or the surroundings model. In particular, an image of the respective current traffic situation in the surroundings of the ego vehicle 110 can be determined for a sequence of consecutive points in time. The image or sequence of images can then be output on the screen 114 of the ego vehicle 110 to assist the driver of the ego vehicle 110 in the longitudinal and / or lateral guidance of the ego vehicle 110.

[0041] Fig. 3 shows an exemplary image 300 relating to a traffic situation in the environment of the ego vehicle 110. The image 300 comprises an (abstract) lane representation 301 of the ego lane 101 as well as lane representations 302, 303 for one or more neighboring lanes 102, 103.

[0042] Furthermore, the image 300 comprises road user symbols 320 for one or more road users 120 (generally objects) in the environment of the ego vehicle 110. Furthermore, a road user symbol 310 for the ego vehicle 110 can be displayed, which is also referred to as an ego symbol and possibly depicts details of the ego vehicle and / or the state of the ego vehicle.

[0043] The image 300 can be generated based on the environment data and / or the environment model such that the ego symbol 310 has a fixed position (e.g., horizontally centered, in the lower area of the image 300). The fixed position can remain unchanged over time. This enables a driver of the ego vehicle 110 to quickly grasp the environmental information presented in an image 300 with respect to the environment of the ego vehicle 110 (since the ego vehicle 110 or the ego symbol 310 can be quickly localized within the image 300).

[0044] In a vehicle 110 operating at SAE Level 3 or higher, the vehicle 110 assumes responsibility for the driving task. If the driver trusts the automated vehicle function or the automated driving mode of the vehicle 110, the driver can effectively use the driving time for other tasks. The effectiveness with which the driver can use the driving time typically increases with increasing driver trust. Thus, trust-building measures can increase the benefits of the automated driving mode.

[0045] Trust can be built up in particular when the driver can recognize that the driving task is being carried out competently and / or reliably by the automated vehicle function (i.e., the automated driving mode). An important aspect in completing the driving task is typically the recognition of relevant objects 120 in the environment of the vehicle 110, i.e., objects 120 that are relevant to vehicle control. The driver's trust in the automated vehicle function can thus be built up in particular by conveying to the driver that the automated vehicle function has recognized the one or more objects 120 that are (typically) considered relevant by the driver of the vehicle 110 and is taking them into account as part of the automated control of the vehicle 110.

[0046] Fig. 1c shows an exemplary driving situation in which the driver 130 of the vehicle 110 manually guides the vehicle 110 longitudinally and / or transversely. The driver 130 of the vehicle 110 observes various objects 121 in the surroundings of the vehicle 110 while driving. The vehicle 110 may include a driver sensor 115, in particular a camera, configured to capture sensor data (also referred to as driver data in this document) relating to the driver 130. The driver data may, in particular, indicate the viewing direction 131 of the driver 130 as a function of time.

[0047] The control unit 111 of the vehicle 110 can further be configured to acquire environmental data relating to the surroundings (as a function of time). The driver data and the environmental data can then be analyzed together to determine which objects 121 are viewed by the driver 130 during manual vehicle control. It is thus possible to determine which objects 121 are typically viewed by the driver 130 (depending on the traffic situation). In other words, the gaze behavior of the driver 130 can be determined.

[0048] Data relating to the actual gaze behavior of the driver 130 and / or relating to the actual gaze behavior of a plurality of different drivers 130 can be used to train an estimation unit to enable the estimation unit to estimate, for a specific driving situation, the one or more objects 121 that would typically be viewed by a driver 130 in the specific driving situation. In particular, an estimation unit can be trained that is configured to predict or estimate, based on the environmental data of the one or more environmental sensors 112 of the vehicle 110, for a specific point in time, which object 121 the driver 130 of the vehicle 110 would view at the specific point in time (if the driver 130 were driving the vehicle 110 manually).

[0049] The estimation unit can, for example, comprise one or more neural networks that have been trained on the basis of training data relating to the gaze behavior of one or more drivers 130. For example, for a specific driving situation, the training data can, on the one hand, comprise the environmental data describing the specific driving situation, and, on the other hand, comprise the driver data describing the direction of gaze 131 (and thus the object 121 viewed by the driver 130).

[0050] Fig. 2a and Fig. 2b shows exemplary components of a neural network 200, in particular a feedforward network. In the example shown, the network 200 comprises two input neurons or input nodes 202, each of which receives a current value of an input variable as input value 201 at a specific time t. The one or more input nodes 202 are part of an input layer 211. In general, the network 200 can be configured to receive an input data set with one or more input values 201. For example, the environmental data (or data derived therefrom) for a specific traffic situation can be passed to the neural network 200 as input values 201.

[0051] The neural network 200 further comprises neurons 220 in one or more hidden layers 212 of the neural network 200. Each of the neurons 220 can have the individual output values of the neurons of the previous layer 212, 211 (or at least a portion thereof) as input values. Processing takes place in each of the neurons 220 to determine an output value of the neuron 220 depending on the input values. The output values of the neurons 220 of the last hidden layer 212 can be processed in an output neuron or output node 220 of an output layer 213 to determine the one or more output values 203 of the neural network 200. In general, the network 200 can be configured to provide output data with one or more output values 203. For example, the estimated viewing direction 131 of the driver 130 for the specific traffic situation can be output as the initial value 203.Alternatively or additionally, an estimated object 121 can be output as the initial value 230, which is presumably viewed by the driver 130 in the specific traffic situation.

[0052] Fig. Figure 2b illustrates the exemplary signal processing within a neuron 220, in particular within the neurons 202 of the one or more hidden layers 212 and / or the output layer 213. The input values 221 of the neuron 220 are weighted with individual weights 222 to determine a weighted sum 224 of the input values 221 in a sum unit 223 (possibly taking into account a bias or offset 227). Using an activation function 225, the weighted sum 224 can be mapped to an output value 226 of the neuron 220. The activation function 225 can, for example, limit the value range. For a neuron 220, for example, a sigmoid function or a hyperbolic tangent (tanh) function or a rectified linear unit (ReLU), e.g., f(x) = max(0, x), can be used as the activation function 225. If necessary, the value of the weighted sum 224 can be shifted with an offset 227.

[0053] A neuron 220 thus has weights 222 and / or, if applicable, an offset 227 as neuron parameters. The neuron parameters of the neurons 220 of a neural network 200 can be learned in a training phase to cause the neural network 200 to approximate a specific function and / or model a specific behavior.

[0054] The training of a neural network 200 can be carried out, for example, using the backpropagation algorithm. For this purpose, in a first phase, a q ten In an epoch of a learning algorithm, corresponding output values 203 are determined at the output of one or more output neurons 220 for the input values 201 at the one or more input nodes 202 of the neural network 200. Based on the output values 203, the error value of an optimization or error function can be determined.

[0055] In a second phase of the q tenDuring the learning algorithm's epoch, the error or error value is backpropagated from the output to the input of the neural network in order to change the neuron parameters of neurons 220 layer by layer. The determined error function at the output can be partially derived for each individual neuron parameter of the neural network 200 in order to determine the extent and / or direction for adjusting the individual neuron parameters. This learning algorithm can be repeated iteratively for a plurality of epochs until a predefined convergence and / or termination criterion is reached.

[0056] In the present context, for example, the environmental data for a specific traffic situation can be transferred as input values 201. The target output values 203 can then indicate which object 121 was actually viewed by the driver 130 in the specific traffic situation. Alternatively or additionally, the actual viewing direction 131 of the driver 130 in the specific traffic situation can be used as the target output value 203. This training data, in particular these training data sets, can be provided for a large number of different traffic situations. The neural network 200 can then be trained in such a way that the object 121 estimated by the neural network 200 and / or the viewing direction 131 estimated by the network 200 corresponds as closely as possible to the respective target output value 203 on average.

[0057] The estimation unit for predicting or estimating the viewing direction 131 of the driver 130 and / or for predicting or estimating the object 121 viewed by the driver 130 can then be used during operation of the automated vehicle function to inform the driver 130 that, in a specific driving situation, the object 121 that would be viewed by the driver 130 has also been detected by the vehicle 110. For example, the object 121 estimated by the estimation unit can be highlighted in the image 300 displayed on the screen 114, as indicated by the symbol 321 in Fig. 3. The highlighting can be achieved by choosing a particular color, a particular intensity, and / or a particular size of the symbol 321.

[0058] Thus, the natural and / or typical gaze behavior of experienced drivers in traffic is used as an input parameter for highlighting a relevant object 121 during automated operation of the vehicle 110. Eye tracking can be used to record the gaze behavior of drivers in traffic. The typical gaze behavior can then be transferred or mapped to a visual representation 300 of the surroundings of the vehicle 110 on a screen 114 of the vehicle 110. Based on the typical gaze behavior of drivers, rules can be derived as to which object 121 is highlighted in which situation.

[0059] For example, the one or more objects 121 on which a driver focuses his gaze for the longest time in an intersection situation (e.g., on a traffic light and / or oncoming traffic in a turning situation) can be highlighted in a pictorial representation 300.

[0060] Fig. 4a shows a flowchart of a (possibly computer-implemented) method 400 for training an estimation unit to enable the estimation unit to estimate, based on environmental data relating to the environment of a motor vehicle 110, a subregion of the environment, in particular an object 121 in the environment, which a driver 130 of the vehicle 110 is likely to observe when the driver 130 is manually driving the vehicle 110. In particular, an estimation unit can be trained that is configured to estimate the one or more objects 121 in the environment of the vehicle 110 that the driver 130 of the vehicle 110 considers relevant when the driver 130 is manually driving the vehicle 110. The object 121 that has the highest probability of being observed by a driver 130 (compared to the other objects 120) can be predicted or estimated.

[0061] The method 400 comprises determining 401 a plurality of training data sets. Each training data set comprises environmental data relating to the environment of a vehicle 110 during a test drive. Furthermore, a training data set comprises information relating to the sub-area of the environment, in particular relating to the object 121 in the environment, of the vehicle 110 that was actually observed by the driver 130 of the vehicle 110 during the test drive. This information can be determined using the sensor data acquired by a driver sensor 115 (in particular using eye tracking).

[0062] The method 400 further comprises training 402 the estimation unit based on the plurality of training data sets. In particular, a neural network 200 of the estimation unit can be trained (as described in connection with the Fig. 2a and Fig. 2b).

[0063] Fig.4b shows a flowchart of an exemplary (possibly computer-implemented) method 410 for increasing the confidence of a user of a motor vehicle 110 in an automated driving mode of the vehicle 110, in which the vehicle 110 is automatically guided longitudinally and laterally (in particular according to SAE Level 3 or higher).

[0064] The method 410 includes, while the vehicle 110 is operating in the automated driving mode, determining 411 environmental data relating to the surroundings of the vehicle 110. The environmental data can be acquired using one or more environmental sensors 112 of the vehicle 110. The environmental data can indicate the traffic situation in which the vehicle 110 is located at a specific point in time. In particular, the environmental data can indicate one or more objects 121 in the surroundings of the vehicle 110 at the specific point in time.

[0065] Furthermore, the method 410 includes estimating 412, using an estimation unit (preliminarily trained based on training data) and based on the surroundings data, a sub-area of the surroundings that a driver 130 of the vehicle 110 would likely observe if the driver 130 were manually driving the vehicle 110, or that an assumed driver manually driving the vehicle would observe. In particular, at least one object 121 in the surroundings of the vehicle 110 can be predicted, identified, or estimated that a driver 130 would likely observe in the present traffic situation.

[0066] The method 410 further includes outputting 413 environmental information relating to the estimated sub-area of the environment, in particular relating to the estimated object 121, to the user of the vehicle 110. For example, the estimated sub-area, in particular the estimated object 121, can be highlighted in a visual representation 300 of the traffic situation. This can convey to the user of the vehicle 110 that the automated driving mode takes the estimated sub-area, in particular the estimated object 121, into account when driving the vehicle 110.

[0067] The measures described in this document can increase the confidence of a user of an automated driving vehicle 110, thereby increasing the comfort and benefits of automated driving.

[0068] The present invention is not limited to the embodiments shown. In particular, it should be noted that the description and figures are intended only to illustrate the principle of the proposed methods, devices, and systems.

Claims

[1] User information system (111) for a motor vehicle (110) with an automated driving mode, in which the vehicle (110) is automatically guided longitudinally and transversely; wherein the user information system (111) is set up while the vehicle (110) is operated in the automated driving mode, - to determine environmental data relating to the environment of the vehicle (110), - by means of an estimation unit of the user information system (111), on the basis of the environment data, to estimate a sub-area of the environment observed by an assumed driver (130) manually driving the vehicle (110); and - outputting environmental information relating to the estimated sub-area of the environment to a user of the vehicle (110). [2] User information system (111) according to claim 1, wherein the user information system (111) is arranged - to detect one or more objects (121) in the surroundings of the vehicle (110) on the basis of the surroundings data; - to estimate by means of the estimation unit which of the one or more objects (121) an assumed driver (130) manually driving the vehicle (110) would observe; and - output environmental information relating to the estimated object (121) to the user of the vehicle (110). [3] User information system (111) according to claim 2, wherein the user information system (111) is designed to output environmental information relating to the estimated object (121) to the user of the vehicle (110) in order to convey to the user that the estimated object (121) is taken into account within the framework of the automated driving mode. [4] User information system (111) according to one of the preceding claims, wherein the user information system (111) is arranged - to determine, on the basis of the environmental data, a traffic situation with one or more other road users (120) in which the vehicle (110) is located; and - to estimate by means of the estimation unit which of the one or more other road users (120) an assumed driver (130) manually driving the vehicle (110) would consider; and - to output environmental information relating to the estimated road user (120) to the user of the vehicle (110). [5] User information system (111) according to one of the preceding claims, wherein the user information system (111) is arranged - to output an image (300) relating to the surroundings, in particular relating to one or more objects (120, 121) in the surroundings, of the vehicle (110) on a screen (114) of the vehicle (110); and - to highlight the estimated sub-area of the surroundings, in particular an estimated object (121) in the surroundings, of the vehicle (110) within the image (300). [6] User information system (111) according to claim 5, wherein - the image (300) comprises, in particular is, a synthetic representation of the surroundings with one or more object symbols (320, 321) for the one or more objects (120, 121) in the surroundings of the vehicle (110); and - the user information system (111) is configured to highlight the object symbol (321) for the estimated object (121) compared to the one or more object symbols (320) for one or more other objects (120) in the surroundings of the vehicle (110). [7] User information system (111) according to one of the preceding claims, wherein - the estimation unit was trained using a machine learning method; and / or - the estimation unit has been trained using training data which, for a large number of different driving situations, indicate the part of the environment which was actually observed by the respective driver (130); and / or - the estimation unit comprises at least one trained neural network (200). [8] User information system (111) according to one of the preceding claims, wherein the estimation unit is designed to take into account a typical gaze behavior of one or more drivers (130) when manually driving a vehicle (110) in order to estimate the sub-area of the surroundings. [9] User information system (111) according to one of the preceding claims, wherein the vehicle (110) is guided longitudinally and transversely in the automated driving mode according to SAE Level 3. [10] User information system (111) according to one of the preceding claims, wherein the environmental data comprise sensor data from one or more environmental sensors (112), in particular from a camera, a lidar sensor and / or a radar sensor, of the vehicle (110). [11] User information system (111) according to one of the preceding claims, wherein the user information system (111) is configured to output the environmental information relating to the estimated sub-area of the environment via a loudspeaker and / or on a screen (114) of the vehicle (110). [12] User information system (111) according to one of the preceding claims, wherein the user information system (111) is arranged - to determine feedback from the user as to whether the estimated part of the environment corresponds to a part of the environment that was actually viewed by the user; and - to train and / or adapt the estimation unit depending on the feedback. [13] Device for training an estimation unit which is configured to estimate, on the basis of environmental data relating to an environment of a motor vehicle (110), a sub-area of the environment which an assumed driver (130) manually driving the vehicle (110) would observe; wherein the device is configured - to determine a plurality of training data sets; each training data set comprises - environmental data relating to the environment of a vehicle (110) during a test drive; and - environmental information relating to the part of the environment of the vehicle (110) that was actually viewed by the driver (130) of the vehicle (110) during the test drive; and - to train the estimation unit based on the large number of training data sets. [14] Method (410) for increasing the confidence of a user of a motor vehicle (110) in an automated driving mode of the vehicle (110), in which the vehicle (110) is automatically guided longitudinally and transversely; wherein the method (410) comprises, while the vehicle (110) is operated in the automated driving mode, - determining (411) environmental data relating to an environment of the vehicle (110); - estimating (412), by means of an estimation unit and on the basis of the environment data, a sub-area of the environment that an assumed driver (130) manually driving the vehicle (110) would observe; and - Outputting (413) environmental information relating to the estimated sub-area of the environment to the user of the vehicle (110). [15] Method (400) for training an estimation unit to enable the estimation unit to estimate, on the basis of environmental data relating to an environment of a motor vehicle (110), a sub-area of the environment that an assumed driver (130) manually driving the vehicle (110) would observe; wherein the method (400) comprises - Determining (401) a plurality of training data sets; wherein a training data set comprises, - environmental data relating to the environment of a vehicle (110) during a test drive; and - information relating to the part of the surroundings of the vehicle (110) that was actually viewed by the driver (130) of the vehicle (110) during the test drive; and - Training (402) the estimation unit based on the plurality of training data sets.

Images