Method, apparatus, computer program and computer program product for verifying a driver assistance system

Abstract

A method for verifying a driver assistance system of a vehicle, wherein the driver assistance system comprises a driver model configured to characterize a predetermined property (P) of a driver of the vehicle based on at least one predetermined input variable, the method comprising the following steps: - providing an input reference data set (REF) for the driver model, which includes at least one reference value for the at least one input variable of the driver model, wherein the at least one reference value satisfies a predetermined detection accuracy requirement, - providing an extended input data set (IN) that represents a statistical distribution of the at least one input variable, wherein the statistical distribution depends on a detection tolerance and / or determination tolerance of the detection sensor.which is used to acquire and / or determine at least one input variable in the vehicle during operation of the driver assistance system; - Acquiring the values ​​of at least one input variable during actual driving operation of the driver assistance system with a reference sensor that has a higher acquisition accuracy than the sensor provided in the vehicle for acquiring the values ​​of at least one input variable; - Providing at least some of the stored values ​​of the input variables as an input reference data set (REF) and / or recording at least some of the stored values ​​of the input variables into the input reference data set (REF).- Determining the driver's characteristic (P) depending on the driver model and the provided extended input data set (IN) and - classifying the determined characteristic (P) as correctly determined or incorrectly determined depending on a reference characteristic corresponding to the input reference data set (REF) and / or - determining a reaction (R) of the driver assistance system depending on the determined characteristic (P) of the driver and classifying the determined reaction (R) as correctly determined or incorrectly determined depending on a reference reaction corresponding to the input reference data set (REF).

Description

The invention relates to a method for verifying a driver assistance system. The invention further relates to a device for verifying a driver assistance system. The invention also relates to a computer program and a computer program product for verifying a driver assistance system. Partially and highly automated driving refer to intermediate steps between assisted driving, in which the driver is supported in the driving task by numerous, often separate driver assistance systems, and autonomous driving, in which the vehicle drives itself and without any input from the driver. In partially automated driving, the driver assistance system takes over steering, acceleration, and braking in specific situations, such as on certain types of roads or within defined speed ranges. However, the driver must continuously monitor the vehicle and be able to take control at any time. One example of such a system is the so-called traffic jam assist, which uses a distance radar and a camera to handle braking, acceleration, and steering in stop-and-go traffic. With highly automated driving, the driver no longer needs to be constantly alert. The system recognizes its own performance limits and actively prompts the driver to take control of the vehicle in such critical situations. The driver must therefore be able to resume driving duties within a certain timeframe. A key requirement for automated or autonomous driving is highly reliable environmental perception by the vehicle. Modern vehicle assistance systems utilize environmental models that, for example, enable the prediction of a wide variety of traffic situations. The verification of these systems, and especially the environmental models, is primarily achieved through driving many kilometers with the active system engaged in diverse traffic environments. As mentioned above, semi-automated and highly automated driving requires ensuring that the driver is able to regain control of the vehicle within a certain time. For modern driver assistance systems for partially or highly automated driving, it is therefore also important to be able to reliably assess the driver's situational awareness. DE 10 2009 009 975 A1 describes a method for determining the attention of a driver during a journey. DE 10 2010 044 449 B4 describes a method for detecting the degree of driving ability of the driver of a motor vehicle using a driver observation camera for observing the driver and an interior camera for observing the interior of the motor vehicle. DE 10 2011 117 850 A1 describes a method for operating a vehicle system of a motor vehicle for determining at least one state variable that describes the state of the driver, in particular the attention and / or fatigue of the driver. DE 10 2015 203 793 A1 describes the collection of data relating to a vehicle's yaw rate during a period of time. The object underlying the invention is to provide a method and a device that makes it possible to reliably verify a driver assistance system for partially and / or highly automated driving and to reduce the effort required for verification. The problem is solved by the features of the independent patent claims. Advantageous embodiments are characterized in the dependent claims. The invention is characterized, according to a first aspect, by a method for verifying a driver assistance system of a vehicle. According to a second aspect, the invention is characterized by a device for verifying a driver assistance system of a vehicle. The device is configured to carry out the method or an embodiment thereof. The driver assistance system comprises a driver model configured to characterize a predetermined characteristic of the driver of the vehicle, depending on at least one predetermined input variable. An input reference data set is provided for the driver model, which includes at least one reference value for the at least one input variable of the driver model. This at least one reference value fulfills a predetermined detection accuracy requirement.Furthermore, an extended input dataset is provided, representing a statistical distribution of the at least one input variable. This statistical distribution depends on the detection tolerance and / or determination tolerance of the sensors used to detect and / or determine the at least one input variable during operation of the driver assistance system. The driver's characteristic is determined based on the driver model and the provided extended input dataset. The determined characteristic is classified as correctly determined or incorrectly determined, depending on a reference characteristic corresponding to the input reference dataset.Alternatively or additionally, a reaction of the driver assistance system is determined depending on the identified characteristic of the driver, and the determined reaction is classified as a correctly determined reaction or an incorrectly determined reaction depending on a reference reaction corresponding to the input reference data set. This allows the driver assistance system to be verified with sufficient reliability. The method described above enables simple validation of the driver assistance system. The driver's characteristics and, if necessary, the response of the driver assistance system can be simulated based on the driver model and the input data set. Securing a driver assistance system that relates to driver characteristics requires that the function of the driver assistance system be verified for a wide variety of different drivers, different driving situations, and different vehicle driving behaviors. By generating and providing the extended input dataset, it is possible to test the driver model for a wide variety of input data values, thus ensuring verification for representative driver coverage. This eliminates the need to verify potential errors or deviations in the input variables caused by different drivers by relying on actual mileage driven by a diverse range of drivers. The input variables are specifically dependent on the characteristics of each individual driver. Furthermore, a large number of different driving situations and / or driving behaviors can be taken into account, so that possible errors or deviations in the input variables caused by driving situations and / or the vehicle's driving behavior do not have to be verified by actual kilometers driven. Furthermore, the simulation based on the extended input data set and the driver model facilitates repeatability of the verification or re-simulation, especially after an adjustment of the driver model. The extended input data set is preferably determined based on the input reference data set and a quality statistic that represents the acquisition tolerance and / or the determination tolerance of the acquisition sensors used for acquiring and / or determining the values ​​for the at least one input variable in the vehicle during operation of the driver assistance system, in particular during normal operation outside of a test operation. In an advantageous embodiment according to the first and second aspects, the classification is based on a comparison of the determined property with the reference property. Preferably, the determined property is classified as correctly determined if it is equal to the reference property or deviates from it within a predefined tolerance range, and as incorrectly determined if it is not equal to the reference property or deviates from it by more than the predefined tolerance range. This allows for a simple decision as to whether the driver assistance system's assessment of the driver's characteristics is sufficiently reliable. In particular, key performance indicators (KPIs) can be easily determined, enabling an evaluation of the degree of fulfillment or improvement of the driver model. In a further advantageous embodiment according to the first and second aspects, the reference property is the same as a determined property where the input reference data set is applied to the driver model. This allows for easy provision of the reference property. In a further advantageous embodiment according to the first and second aspects, during actual driving operation of the driver assistance system, the values ​​of the input variables are acquired using a reference sensor system that has a higher acquisition accuracy than the sensors provided in the vehicle for acquiring the values. At least some of the stored values ​​of the input variables are provided as an input reference data set and / or included in the input reference data set. To determine a driver's characteristics while driving, such as attentiveness or fatigue, the driver's gaze direction can be recorded using an eye tracker. Eye trackers are devices and systems that record eye movements and allow for their analysis. To provide the input reference data, the gaze direction can be determined based on supplementary camera recordings that offer higher resolution, better lighting, and / or greater mechanical stability, among other advantages. This also allows for the verification of the gaze direction and its accuracy as determined by the eye tracker. A similar approach can be used to determine reference values ​​for the driver's position and / or head pose.Optionally, the values ​​of these input variables, which are recorded by the vehicle's intended sensors, can also be verified. In a further advantageous embodiment according to the first and second aspects, the values ​​of the at least one input variable are recorded for different driving situations and / or different vehicle driving behaviors and / or different driver behaviors. Driver behavior here includes, in particular, the driver's driving style. Advantageously, this enables the provision of an input reference data set for a multitude of different boundary conditions with regard to driving situations, driver behavior, and vehicle driving behavior. Preferably, data on the respective driving situation and / or the respective driver behavior and / or the respective vehicle driving behavior are also recorded and stored during actual driving operation. The input reference data are thus so-called "ground truth data".This means that the input reference dataset represents a dataset where, in particular through comparison, it has been determined that the data recorded during real-world driving corresponds to the data determined by the driver model or the driver assistance system and / or leads to the same result. The input reference data thus fulfills the requirements for the respective driving situation, the respective vehicle driving behavior, and real-world driver behavior. The respective reference values ​​of the at least one input variable of the input reference dataset are preferably to be regarded as the respective reference for a specific driving situation with a specific driver behavior, in particular a specific driving style, and a specific vehicle driving behavior. In a further advantageous embodiment according to the first and second aspects, the driver's characteristics are determined depending on a given driving situation and / or a given driving behavior of the vehicle. In particular, information about the respective driving situation and the respective driving behavior of the vehicle can be incorporated into the driver model for determining the characteristics. Depending on which driver characteristic is being assessed, the evaluation can also depend on a driving situation or traffic situation and / or driving behavior. For example, when assessing attention, specific areas and / or objects in the vehicle's surroundings can be identified for which attention is to be evaluated. Alternatively or additionally, a change in direction of travel can be taken into account when assessing attention. In a further advantageous embodiment according to the first and second aspects, the driver's characteristics are determined for a multitude of given driving situations and / or a multitude of given driving behaviors of the vehicle. In a further advantageous embodiment according to the first and second aspects, the characteristic of the driver includes a condition and / or a behavior of the driver. In a further advantageous embodiment according to the first and second aspects, the driver's characteristic represents the driver's attention. A reliable assessment of the driver's attention can be advantageously used for driver assistance systems for partially and highly automated driving. This increases the safety of the driver assistance functions and prevents their misuse, since it is always possible to reliably monitor whether the driver is appropriately attentive to the situation during a journey, especially during partially automated driving. For the driver to have a truly accurate situational awareness, it is necessary that the driver perceives the objects in the environment, understands their significance, and accurately predicts changes in the environment and the future state of the objects for a sufficient period of time.Such an assessment of situational awareness is therefore often based on an assessment of what the driver is focusing his attention on, and a recognition of driver intent depending on the driver's current behavior. In a further advantageous embodiment according to the first and second aspects, the driver's characteristic represents driver fatigue. This fatigue can be determined based on head posture and / or blink rate. In a further advantageous embodiment according to the first and second aspects, the driver model comprises a first input variable that is representative of a direction of view of the driver, and / or a second input variable that is representative of a head pose of the driver, and / or a third input variable that is representative of a head position of the driver. The head pose can include a tilt and / or yaw angle of the head. A gaze direction can include a gaze vector and / or nose vector. According to a third aspect, the invention is characterized by a computer program, wherein the computer program is configured to carry out the method for verifying a driver assistance system or an optional embodiment of the method. According to a fourth aspect, the invention is characterized by a computer program product comprising an executable program code, wherein the program code, when executed by a data processing device, performs the method for verifying a driver assistance system or an optional embodiment of the method. The computer program product includes, in particular, a medium readable by the data processing device on which the program code is stored. Exemplary embodiments of the invention are explained in more detail below with reference to the schematic drawing. Figure 1 shows a flowchart for verifying a driver assistance system. Figure 1 shows a flowchart of a program for verifying a driver assistance system. The program can be executed by a device for verifying a driver assistance system. The device comprises, in particular, a processing unit, a program and data memory, and, for example, one or more communication interfaces. The program and data memory and / or the processing unit and / or the communication interfaces can be integrated into a single unit and / or distributed across multiple units. The program and data storage of the device contains, in particular, the program for verifying a driver assistance system. The driver assistance system to be verified includes at least one driver model that is trained to characterize at least one predefined characteristic P of the driver, depending on predefined input variables. The driver assistance system is a system that evaluates the driver's state and / or behavior in order to execute one or more driver assistance functions. For example, the driver assistance system is trained to execute a driver assistance function based on the determined driver state and / or behavior, such as issuing a warning and / or actively intervening in vehicle control, for example, initiating braking. The driver assistance system, for example, includes a driver model to assess the driver's attention and / or fatigue. For instance, to control a driver assistance function, the system may require information about whether the driver looked at a specific object within the last second. Furthermore, instead of a simple yes / no function, it can differentiate between, for example, "definitely not seen," "possibly seen," or "very likely seen." The program can therefore be used, for example, to verify whether the driver assistance system performs an attention assessment and / or fatigue assessment for the driver with sufficient reliability. The program starts in step S1, in which variables can be initialized if necessary. The program continues in step S3. In step S3, an input reference data set REF is provided for the at least one driver model, containing reference values ​​for the respective input variables of the driver model. Providing this data set can, for example, involve retrieving the input reference data set REF from a database. Preferably, the input reference data set REF is determined and stored in the database prior to the actual verification simulation. In the case of attention assessment and / or fatigue assessment, the input variables may include, for example, the driver's gaze direction and / or head pose and / or head position. The head pose can include a tilt and / or yaw angle of the head. A gaze direction can include a gaze vector and / or nose vector. The program continues in step S5. In step S5, an extended input data set IN is provided, representing a statistical distribution of the at least one input variable. This statistical distribution depends on the acquisition tolerance and / or determination tolerance of the sensors used to acquire and / or determine the at least one input variable. Providing this data set can, for example, involve retrieving the extended input data set IN from the database or another database. Preferably, the extended input data set IN is determined prior to the actual verification simulation and stored in the corresponding database. The extended input dataset IN, for example, comprises a multitude of one-dimensional input data arrays, where the number of elements in each input data array is determined by the number of input variables, and each one-dimensional input data array contains one value for the respective input variable. The multitude of input data arrays is used, in particular, to represent the statistical distribution of the values ​​of the different input variables. The program continues in step S7. In step S7, the driver's property P is determined based on the driver model and the provided extended input data set IN. In the case of attention scoring, for example, a visual field area and / or a primary visual field and / or peripheral visual field of the driver are determined based on the input variables specifically mentioned above. The driver's characteristic P is also determined in step S7, for example, depending on a driving situation or a traffic situation and / or the vehicle's driving behavior. For example, the driver's attention and / or fatigue are determined depending on the driving situation. This involves identifying areas and / or objects to which the driver should and should not focus their attention. The attention assessment thus allows at least a statement about the probability that an object to be viewed and / or an area to be observed lies within the driver's field of vision, particularly within their primary field of vision. Depending on the given areas or the objects being viewed, it takes a person longer to fully perceive them. Humans perceive an object roughly within 1 to 10 seconds. Attention assessment therefore also considers, for example, how long and / or how frequently a driver focuses their primary field of vision on a specific object or area. Alternatively or additionally, the driver's attention and / or fatigue is determined based on the vehicle's driving behavior. Modern driver assistance systems for monitoring driver attention include, for example, a camera sensor that determines the driver's head pose, gaze direction, and / or position, for instance, based on facial features. The camera sensor can utilize a head model of the driver for this purpose. This head model can be based on specific anatomical or physiological features such as the angle of the eyes, the corners of the mouth, the tip of the nose, and their relationship to one another. The assessment of the driver's attention can then be based on a detected head movement within a fixed vehicle model. However, a problem arises from the fact that, due to the high dynamism of road traffic, the driver's attention is often misjudged. A camera sensor can be used to determine the input parameters, which are determined based on various facial features. The dynamics of road traffic or the driving behavior of the vehicle is determined, for example, depending on road data that are representative of a road course lying in the direction of travel of the vehicle, and / or vehicle data, for example steering angle, yaw rate and so on. Attention and / or fatigue are therefore also determined based on factors such as road data and / or vehicle data. The program continues in step S9 or S11. In step S9, the determined property P is classified as correctly determined or incorrectly determined, depending on a reference property corresponding to the input reference data set REF. Preferably, the classification is based on a comparison of the determined property P with the reference property. The reference property is, in particular, the same as a determined property where the input reference data set REF is applied to the driver model. In the alternative or additional step S11, a reaction R of the driver assistance system is determined depending on the determined characteristic P of the driver, and the determined reaction R is classified as a correctly determined reaction or an incorrectly determined reaction depending on a reference reaction corresponding to the input reference data set REF. The driver property P is determined, for example, for at least a portion of the input data arrays of the extended input data set IN, and the program continues in step 7 after step S9 or step S11 until a sufficient number of input data points have been evaluated. After evaluating a sufficient number of input data points, preferably after evaluating the entire extended input data set IN, the program terminates in step S13. The determination of the input reference data set REF is preferably carried out independently of the execution of the program for verifying the driver assistance system. To provide the input reference data set REF, the values ​​of the input variables are acquired and stored, for example, during actual driving operation of the driver assistance system, using a reference sensor system that has a higher detection accuracy than the sensors installed in the vehicle for acquiring the values. At least some of the stored values ​​of the input variables are provided as an input reference data set and / or included in the input reference data set. The input variables of the driver model include, for example, the driving situation and / or the vehicle's driving behavior, the driver's attribute P (in particular, the driver's state and / or behavior), and the response R of the driver assistance system. Preferably, this data is acquired using additional sensors and is not the data determined by the driver assistance system. This allows verification of whether the input reference data, when applied to the driver model, leads to the same result as that obtained or observed in real-world driving. The input reference data are therefore so-called "ground truth data", meaning that the input reference data set REF represents a data set in which the requirements for the respective driving situation, the respective driving behavior of the vehicle and a real behavior of the driver, in particular a driving style of the driver, are met. The accuracy of the input values ​​of the driver model is verified, for example, depending on the associated stored driving situation and / or the associated stored driving behavior and / or the driver's driving style and the associated stored driver characteristic. In particular, the input reference data set REF can comprise a sub-reference data set for the respective driving situation and / or the respective driving behavior and / or the driving style of the driver. The determination of the extended input data set IN is preferably also carried out independently of the execution of the program for verifying the driver assistance system. The extended input data set IN includes, for example, values ​​of the at least one input variable, which represents a statistical distribution of a detection tolerance and / or determination tolerance of the detection sensor with which the at least one input variable is detected and / or determined. The tolerances can vary depending on the driving situation, vehicle behavior, and / or driver driving style. Therefore, the extended input data set IN contains, for example, subsets for the different driving situations, vehicle behavior, and / or driver driving styles. Reference symbol list REF Input reference data set IN Extended input data set P Driver characteristic R Driver assistance system response S1... S13 Program steps

Claims

A method for verifying a driver assistance system of a vehicle, wherein the driver assistance system comprises a driver model configured to characterize a predetermined property (P) of a driver of the vehicle based on at least one predetermined input variable, the method comprising the following steps: - providing an input reference data set (REF) for the driver model, which includes at least one reference value for the at least one input variable of the driver model, wherein the at least one reference value satisfies a predetermined detection accuracy requirement, - providing an extended input data set (IN) that represents a statistical distribution of the at least one input variable, wherein the statistical distribution depends on a detection tolerance and / or determination tolerance of the detection sensor.which is used to acquire and / or determine at least one input variable in the vehicle during operation of the driver assistance system; - Acquiring the values ​​of at least one input variable during actual driving operation of the driver assistance system with a reference sensor that has a higher acquisition accuracy than the sensor provided in the vehicle for acquiring the values ​​of at least one input variable; - Providing at least some of the stored values ​​of the input variables as an input reference data set (REF) and / or recording at least some of the stored values ​​of the input variables into the input reference data set (REF).- Determining the driver's characteristic (P) depending on the driver model and the provided extended input data set (IN) and - classifying the determined characteristic (P) as correctly determined or incorrectly determined depending on a reference characteristic corresponding to the input reference data set (REF) and / or - determining a reaction (R) of the driver assistance system depending on the determined characteristic (P) of the driver and classifying the determined reaction (R) as correctly determined or incorrectly determined depending on a reference reaction corresponding to the input reference data set (REF). Method according to claim 1, wherein the classification depends on a comparison of the determined property (P) with the reference property. Method according to claim 1 or 2, wherein the reference property is equal to a determined property where the input reference data set (REF) is applied to the driver model. Method according to one of the preceding claims, wherein the values ​​of the input variables are recorded for different driving situations and / or different driving behavior of the vehicle and / or different driver behavior. Method according to one of the preceding claims, wherein the characteristic (P) of the driver is determined depending on a given driving situation and / or a given driving behavior of the vehicle. Method according to one of the preceding claims, wherein the characteristic (P) of the driver is determined for a plurality of predetermined driving situations and / or a plurality of predetermined driving behaviors of the vehicle. Method according to one of the preceding claims, wherein the characteristic (P) of the driver comprises a condition and / or a behavior of the driver. Method according to any of the preceding claims, wherein the driver's property (P) represents the driver's attention. Method according to any of the preceding claims, wherein the driver characteristic (P) represents driver fatigue. Method according to one of the preceding claims, wherein the driver model comprises a first input variable that is representative of a direction of view of the driver, and / or a second input variable that is representative of a head pose of the driver, and / or a third input variable that is representative of a head position of the driver. Device for verifying a driver assistance system, wherein the driver assistance system comprises a driver model configured to characterize a predetermined property (P) of the driver depending on at least one predetermined input variable, and the device is configured to perform a method according to one of claims 1 to 10. Computer program for verifying a driver assistance system, wherein the computer program is configured to perform a method according to one of claims 1 to 10 when executed on a data processing device. A computer program product comprising executable program code, wherein the program code, when executed by a data processing device, performs the method according to any one of claims 1 to 10.

Images