Method and device for identifying a malfunction in a surroundings model of an automated driving function

Abstract

Provided is a method for identifying a malfunction in a surroundings model that is used by an automated driving function of a motor vehicle, the method comprising: determining a first deviation between a target trajectory determined by the surroundings model and an actual trajectory travelled by the motor vehicle and / or a second deviation between a course of a road determined by the surroundings model and a course of a road determined by camera software; and identifying the malfunction based on the first and / or the second deviation.

Description

[0001] METHOD AND DEVICE FOR DETECTING A MALFUNCTION OF AN ENVIRONMENTAL MODEL OF AN AUTOMATED DRIVING FUNCTION

[0002] The present disclosure relates to a method for detecting a malfunction of an environment model of an automated driving function. Additionally or alternatively, a data processing device is provided which is designed to at least partially carry out the method. Additionally or alternatively, an automated motor vehicle which at least partially comprises the data processing device is provided. Additionally or alternatively, a computer program is provided which comprises instructions which, when the program is executed by a computer, cause the computer to at least partially carry out the method. Additionally or alternatively, a computer-readable medium is provided which comprises instructions which, when the instructions are executed by a computer, cause the computer to at least partially carry out the method.

[0003] In an automated driving function of a motor vehicle, it is disruptive for the driver if the vehicle incorrectly detects the geometry of a road or its course, resulting in the vehicle not following the road as intended, e.g., due to steering errors. In this case, driver intervention may be necessary, taking over control of the vehicle's lateral and / or longitudinal guidance. If frequent intervention is necessary, this reduces the driver's confidence in the automated driving function.

[0004] When developing automated driving functions, special attention is therefore paid to situations in which the vehicle steers and does not follow the real road course.

[0005] A test driver who detects such a malfunction of the vehicle notes it and manually starts a data recording to record the incident.

[0006] This data recording can be analyzed by development, and it can be determined which part of the automated driving function is responsible for the malfunction. However, many manual steps are required for data recording and analysis.

[0007] However, the test driver generally cannot determine the reason for the faulty behavior or which part of the automated driving function triggered this faulty behavior.

[0008] The cause of the malfunction is traditionally determined by developers through manual visual analysis of the recorded data. This involves visually comparing the road geometry calculated by the environment model with reality (driven trajectory or camera images).

[0009] It is also known to use the trajectory driven by a driver as training data for the automated driving function (so-called behavioral cloning).

[0010] Against the background of this prior art, the object of the present disclosure is to provide a device and a method which are each suitable for enriching the prior art described above.

[0011] The problem is solved by the features of the independent claim. The subclaims contain preferred developments of the invention.

[0012] According to this invention, the object is achieved by a method for detecting a malfunction of an environment model that is used by an automated driving function of a motor vehicle. The method comprises determining a first deviation between a target trajectory determined by the environment model and an actual trajectory driven by the motor vehicle. Additionally or alternatively, the method comprises determining a second deviation between a road course determined by the environment model and a road course determined by camera software. Furthermore, the method comprises detecting the malfunction based on the first and / or the second deviation. The method described above can be a computer-implemented method, i.e. one, several or all steps of the method can be carried out by a computer or a data processing device.

[0013] An automated driving function can be understood as a function of the motor vehicle which is designed to take over certain parts of the driving task or the entire driving task, possibly together with other automated driving functions, at least temporarily.

[0014] The environment model can be a so-called road environment model. The environment model can be determined based on sensor data from several, particularly different, sensors in the vehicle. For this purpose, the sensor data can be merged. The road environment model can include information about the road course and the desired trajectory of the vehicle.

[0015] A trajectory can be understood as a path along which a motor vehicle is supposed to travel (so-called target trajectory) or is actually traveling (so-called actual trajectory). In addition to position information, the trajectory can also contain a temporal component, i.e., when the motor vehicle is supposed to travel or is where.

[0016] The two options for determining the deviation described above have the common inventive idea of ​​automatically detecting and, if necessary, subsequently analyzing any malfunction of the automated driving function in the area of ​​the (road) environment model.

[0017] The explanations for the two options can therefore be combined and, as far as technically reasonable, apply equally to both options.

[0018] In addition to reducing costs and accelerating the development of automated driving functions, the process also offers the opportunity to improve the quality of automated driving functions through the automation of manual processes and data-driven development. Possible further developments of the above process are described in detail below.

[0019] Determining the first deviation between the target trajectory determined by the environment model and the actual trajectory driven by the motor vehicle can comprise determining a position and / or a curvature of a center line of a roadway using the environment model as the target trajectory, and determining the first deviation based on a deviation between the position and / or the curvature of the center line and a position and / or a curvature of the actual trajectory.

[0020] Determining the second deviation between the road course determined by the environment model and the road course determined by the camera software can be determining a position and / or a curvature of a lane marking and / or a center line of a roadway using the environment model as the road course, determining a position and / or curvature of the lane marking and / or a center line of the roadway using the camera software as the road course, and determining the second deviation based on a deviation between the position and / or the curvature of the lane marking and / or the center line determined by means of the environment model and the position and / or the curvature of the lane marking and / or the center line determined by means of the camera software.

[0021] The method comprises determining the existence of a predetermined environmental situation and recognizing that, despite the second deviation, there is no malfunction.

[0022] The first and / or second deviation can be determined during a journey in the motor vehicle (i.e., online) and / or after the journey using a data processing device external to the motor vehicle (i.e., offline). Data used by the automated driving function can be stored in a ring buffer if the first and / or second deviation is determined during the journey (i.e., online).

[0023] The data stored in the ring buffer can be sent from the motor vehicle to the data processing device external to the motor vehicle when the malfunction is detected based on the first and / or the second deviation.

[0024] The malfunction can be detected if the first and / or the second deviation exceeds a respective predetermined limit.

[0025] The method described above can be summarized in other words and in a more concrete embodiment as described below as non-limiting to the present disclosure.

[0026] The aim is to avoid manual steps in the analysis and quality assessment of the road geometry calculated by the automated driving function. The quality assessment can take place both online and offline. An online quality assessment (self-monitoring) can be used for the automated detection of deviations during the journey. This allows malfunctions to be detected automatically, the driver to take over and / or data to be recorded automatically for offline analysis. Since the recording of problematic situations is automated, the data can be recorded not only by test drivers with special equipment, but also in customer vehicles. The increased number of vehicles that can record the data can make it easier to record rare incidents or malfunctions. The recorded data can be recorded automatically or.be analyzed automatically and / or included in a training data collection.

[0027] The disclosure provides several measures for automatically detecting and analyzing malfunctions of an automated driving function in the area of ​​the road environment model. In one aspect, the disclosure determines a deviation between the road course determined by the automated driving function (road environment model) and the actual trajectory.

[0028] For example, the lateral offset between a center line calculated by the environment model and the actual trajectory can be used as a deviation. Alternatively, or in addition, a deviation based on the steering angle can also be calculated.

[0029] The deviation can be determined both online in the vehicle and offline.

[0030] The environment model can comprise several submodels (lane fusion, crowd trajectories, HD map). The invention allows the individual models to be evaluated.

[0031] When determining the deviation online in the vehicle, if the deviation is too large, data can be automatically recorded for further analysis.

[0032] Data recording can also occur when other critical values ​​are detected, e.g. excessive curvatures, particularly small or large track widths, very short or unstable detections of the road markings by the camera, etc.

[0033] In a further aspect of the disclosure, to which the above description of the first aspect applies mutatis mutandis, ie can be combined therewith, a geometry calculated by the road environment model (ie an output of the road environment model) can be compared with a geometry of a detection of a lane marking and / or estimated centerline provided by camera software (ie an input of the road environment model).

[0034] There can be at least two reasons for a deviation. Firstly, current camera detections may be faulty, e.g., the camera may mistake tar seams or shadow edges for lane markings and / or incorrectly detect the geometry (particularly a curvature) of the lane markings. The automated driving function may therefore include a detection system that recognizes certain special situations (i.e., predetermined scenarios) in which camera misdetections are known to frequently occur. In this case, current camera detections can be discarded and not taken into account when calculating the road environment model. In this case, the deviation may be intentional.

[0035] On the other hand, despite correct input data, deviations can also arise due to incorrect processing in the environment model. This can be an unwanted deviation, indicating a possible processing error in the environment model.

[0036] The identified special situations (scenarios) can serve as a criterion to determine the reason for the deviation.

[0037] Another indication of malfunction in the road environment model can be the driver taking over control (so-called driver takeover). The driver can be prompted to take over control of the vehicle, e.g., if self-monitoring detects an error, or the driver can take over control on their own initiative, e.g., if the driver detects a malfunction.

[0038] Driver takeover can be used as a trigger for data recording. It can also, in addition or alternatively, serve as a criterion for analyzing the cause of the vehicle's malfunction.

[0039] If the geometry of the road environment model and the camera detections match at the time of driver takeover, this may indicate a fault in the camera software. However, if a discrepancy exists, this may indicate a fault in the road environment model. Data recording can use a ring buffer so that when recording starts, the period prior to the incident or malfunction is also recorded. This allows for analysis of the onset of the discrepancy.

[0040] The recorded data can be transferred from the vehicle, e.g., the customer's vehicle, to a cloud (e.g., the vehicle manufacturer's). This allows for the compilation of statistics on anomalies, particularly global statistics, from all and / or specific customer vehicles.

[0041] The method can therefore be used for error analysis. Statistical evaluation can be used to focus on the most common error patterns.

[0042] Automated recording can be used to identify geographical hotspots of misconduct.

[0043] The method can be used to detect a faulty component of the automated driving function (e.g. camera software, lane marking fusion, localization, crowd trajectory map, HD map, etc.).

[0044] Object and / or obstacle information can also be included to determine reasons for the detour or the deviating trajectory.

[0045] The recorded data can be automatically included in a training database for the automated driving function and / or can be provided to a supplier of the automated driving function and / or the camera software for the purpose of improving the automated driving function or the camera software.

[0046] Furthermore, a data processing device is provided, which comprises means for carrying out the method described above. The data processing device can also be referred to as a data processing system or a data processing device. The data processing device can be configured to be installed at least partially in and / or on an automated motor vehicle and / or to be at least partially part of a cloud.

[0047] The data processing device can be part of a driver assistance system or an automated driving function, or it can represent and / or execute this. The data processing device can have an electronic control unit (ECU). The data processing device can have an intelligent processor-controlled unit which can communicate with other modules, for example via a central gateway (CGW), and which can optionally form the vehicle's on-board network via field buses such as the CAN bus, LIN bus, MOST bus and FlexRay or via automotive Ethernet, for example together with telematics control units. It is conceivable that the control unit controls functions relevant to the driving behavior of the motor vehicle, such as engine management, power transmission, braking system and / or tire pressure monitoring system. In addition, driver assistance systems, such as a parking assistant, adaptive cruise control (ACC) or adaptive cruise control (ADC) can be integrated.adaptive cruise control), a lane keeping assistant, a lane change assistant, traffic sign recognition, light signal recognition, a start-off assistant, a night vision assistant, an emergency braking assistant and / or an intersection assistant are controlled by the control unit.

[0048] It is conceivable that the data processing device controls automated driving of the automated motor vehicle at least partially and / or temporarily based on an output of an environment model.

[0049] What has been described above with reference to the methods also applies analogously to the data processing device and vice versa.

[0050] Furthermore, an automated motor vehicle can be provided which has the data processing device described above.

[0051] The motor vehicle can be a passenger car, in particular an automobile. The automated motor vehicle can be configured to assume longitudinal and / or lateral guidance during automated driving of the motor vehicle, at least partially and / or at least temporarily, by means of the automated driving function. Automated driving can be controlled at least partially and / or temporarily by the data processing device using the automated driving function.

[0052] Automated driving can be achieved in such a way that the vehicle moves (largely) autonomously.

[0053] The motor vehicle may be a motor vehicle with autonomy level 0, i.e. the driver assumes the dynamic driving task, even if supporting systems (e.g. ABS or ESP) are present.

[0054] The motor vehicle may be a Level 1 autonomy motor vehicle, i.e. it may have certain driver assistance systems that support the driver in operating the vehicle, such as adaptive cruise control (ACC).

[0055] The motor vehicle may be a motor vehicle of autonomy level 2, i.e. be so partially automated that functions such as automatic parking, lane keeping or lateral guidance, general longitudinal guidance, acceleration and / or braking are taken over by driver assistance systems.

[0056] The motor vehicle can be a Level 3 autonomy vehicle, meaning it is so conditionally automated that the driver does not need to continuously monitor the vehicle system. The vehicle performs functions such as activating the turn signal, changing lanes, and / or keeping in lane independently. The driver can attend to other tasks but will be prompted by the system to take over control within a pre-warning period if necessary.

[0057] The motor vehicle can be a Level 4 autonomy vehicle, meaning it is so highly automated that the vehicle system permanently assumes control of the vehicle. If the system can no longer handle the driving tasks, the driver can be requested to take over.

[0058] The motor vehicle can be a Level 5 autonomy vehicle, meaning it is so fully automated that the driver is not required to perform the driving task. No human intervention is required other than setting the destination and starting the system.

[0059] What has been described above with reference to the methods and the data processing device also applies analogously to the motor vehicle and vice versa.

[0060] Furthermore, a computer program is provided. The computer program is characterized in that it comprises instructions that, when executed by a computer, cause the computer to at least partially execute the described method.

[0061] A program code of the computer program may be in any code, in particular in a code suitable for motor vehicle controls.

[0062] Furthermore, a computer-readable medium, in particular a computer-readable storage medium, is provided. The computer-readable medium is characterized in that it comprises instructions that, when the program is executed by a computer, cause the computer to at least partially execute the method described above.

[0063] This means that a computer-readable medium can be provided that contains a computer program as defined above. The computer-readable medium can be any digital data storage device, such as a USB stick, a hard drive, a CD-ROM, an SD card, or an SSD card. The computer program does not necessarily have to be stored on such a computer-readable storage medium to be provided to the motor vehicle; it can also be obtained externally via the Internet or otherwise.

[0064] The above description with regard to the methods, the data processing device and the automated motor vehicle also applies analogously to the computer program and the computer-readable medium and vice versa.

[0065] An embodiment is described below with reference to Figures 1 to 3.

[0066] Fig. 1 shows a schematic flow diagram of a method for detecting a malfunction of an environment model used by an automated driving function of a motor vehicle,

[0067] Fig. 2 shows schematically a first plan view of a road surface to explain the determination of a first deviation between a target trajectory determined by the environment model and an actual trajectory driven by the motor vehicle, and

[0068] Fig. 3 schematically shows a second plan view of a roadway plane to explain the determination of a first deviation between a target trajectory determined by the environment model and an actual trajectory driven by the motor vehicle.

[0069] As can be seen from Figure 1, the method for detecting the malfunction of the environment model used by the automated driving function of the motor vehicle essentially comprises two steps S1 and S2.

[0070] In a first step of the method, a first deviation is determined between a target trajectory determined by the environment model and an actual trajectory driven by the motor vehicle. Additionally or alternatively, a second deviation is determined between a road course determined by the environment model and a road course determined by camera software. Some concrete implementation options are described in detail below with reference to Figures 2 to 3. This does not represent a comprehensive description of the implementation options, but serves merely as an explanatory guide.

[0071] Determining the first deviation between the target trajectory determined by the environment model and the actual trajectory driven by the motor vehicle can comprise determining a position of a center line of a roadway using the environment model as the target trajectory and determining the first deviation based on a deviation between the position of the center line and the position of the actual trajectory.

[0072] This is shown as an example for a global coordinate system in Figure 2. The driven trajectory or actual trajectory 1, which is determined via an odometry of the motor vehicle and is shown with a solid line in Figure 2, is compared with data from the environment model, here a center line t1, t2 at different points in time, each shown with a dashed line, in the global coordinate system. For this purpose, the odometry data, in particular position information, and the center lines t1, t2 from a time interval are entered into the global coordinate system. In Figure 2, this is shown as an example for two different points in time t1 and t2. The respective deviations between the real trajectory or actual trajectory 1 and the calculated center line t1, t2 of the environment model at different points in time at the location orPosition at which the motor vehicle was located at the time the center line was determined. The deviation can be the vertical distance between the actual trajectory 1 and the calculated center line t1, t2 or, as shown in Figure 2, the shortest distance d1 or d2 between the actual trajectory 1 and the calculated center line t1, t2. The distance d1, d2 between these can at least be included in the calculation or determination of the first deviation. Additionally or alternatively, a curvature of the center line and a curvature of the actual trajectory can be determined at the respective positions of the motor vehicle and the difference between the curvatures can at least be included in the calculation or determination of the first deviation. A steering angle of the motor vehicle can be used for this purpose, i.e.The respective curvature of the center line t1, t2 can be determined at different positions and compared with the actual steering angle of the vehicle at the respective position. It is also conceivable that the center lines of the environment model are preprocessed, e.g., by creating a single global center line from the center lines of several time steps or points in time t1, t2, and determining the deviation, e.g., the maximum distance between the actual trajectory and this global center line.

[0073] As shown in Figure 3, a coordinate system of the motor vehicle, i.e. a relative coordinate system, can also be used to determine the first deviation. Analogous to determining the deviation using the global coordinate system, as described above with reference to Figure 2, the current position 2 or actual position of the motor vehicle can be compared with a current target position of the motor vehicle, which lies on the center line t1, and in this way the distance d1 can be determined. This can be done continuously. Determining the first deviation using the relative coordinate system can be particularly suitable if it is determined online while driving and can be done continuously. The center line t1 used for this can be determined in the same journal or in an earlier journal.This means that the current position 2 of the vehicle can be compared with a target position determined by the environment model at a previous time (e.g., t=0). Also analogous to the method described above, the curvature of the center line t1 and the current steering angle can be compared.

[0074] It is also conceivable that the center line t1, t2 is determined by both the environment model and the camera software. In this case, the position and / or curvature of the center lines t1, t2 can be determined using both the camera software and the environment model, and a deviation between the position and / or curvature of the center lines can be determined. The deviation between the center lines, e.g. their lateral offset (see above), can at least be included in the calculation or determination of the second deviation. It would also be conceivable to use a road marking in this way, in addition to or as an alternative to the center line. In this case, the position and / or curvature of a road marking can be determined using both the camera software and the environment model, and a deviation between the position and / or curvature of the road markings can be determined. The deviation between the road markings, e.g.their lateral offset (see above) can at least be included in the calculation or determination of the second deviation.

[0075] In a second step S2 of the method, the malfunction is detected based on the first and / or the second deviation, wherein the malfunction is detected if the first and / or the second deviation exceeds a respective predetermined limit value. It is conceivable that the second deviation is not taken into account if it is determined that a predetermined environmental situation exists. This means that the automated driving function can be provided with a detection system that detects predetermined environmental situations or special situations (i.e. predetermined scenarios) in which camera false detections are known to occur frequently. In this case, current camera detections can be discarded and not taken into account when calculating the environmental model. In this case, the deviation can be intentional.In such a case, it can therefore be recognized that despite the second deviation exceeding the limit value, there is no malfunction.

[0076] The above-described determination of the first and / or second deviation can take place during a journey in the motor vehicle and / or after the journey using a data processing device external to the motor vehicle. The same applies to the above-described detection of the malfunction based on the first and / or second deviation. Data used by the automated driving function can be stored in a ring buffer if one or both of these steps takes place during the journey. The data stored in the ring buffer can be sent from the motor vehicle to the data processing device external to the motor vehicle if the malfunction is detected based on the first and / or second deviation.Additionally or alternatively, the data can be recorded with a recording device, in particular a data logger, during the entire journey or parts thereof, regardless of whether a malfunction is detected, and / or sent to the data processing device external to the motor vehicle, in particular streamed to a backend or a cloud.

[0077] List of reference symbols

[0078] 1 Actual trajectory 2 Current position of the motor vehicle t1 , t2 Target trajectory at different times d1 , d2 Distance between actual trajectory and the respective target trajectory at the different times

[0079] S1 , S2 steps of the procedure

Claims

Patent claims 1. Method for detecting a malfunction of an environment model used by an automated driving function of a motor vehicle, wherein the method comprises: - Determining a first deviation between a target trajectory (t1, t2) determined by the environment model and an actual trajectory (1) driven by the motor vehicle and / or a second deviation between a road course determined by the environment model and a road course determined by camera software, and - Detecting the malfunction based on the first and / or the second deviation.

2. Method according to claim 1, wherein determining the first deviation between the target trajectory (t1 , t2) determined by the environment model and the actual trajectory (1 ) driven by the motor vehicle comprises: - Determining the position and / or curvature of a roadway centerline using the environment model as the target trajectory (t1, t2), and - Determining the first deviation based on a deviation between the position and / or curvature of the midline and a position and / or curvature of the actual trajectory (1 ).

3. Method according to claim 1 or 2, wherein determining the second deviation between the road course determined by the environment model and the road course determined by the camera software comprises: Determining the position and / or curvature of a lane marking and / or a center line of a roadway using the environment model as the road alignment, - Determining the position and / or curvature of the lane markings and / or a center line of the roadway using camera software as the road alignment, and - Determining the second deviation based on a deviation between the position and / or curvature of the lane marking and / or center line determined by the environment model(s) and the position and / or curvature of the lane marking and / or center line determined by the camera software(s). Method according to any one of claims 1 to 3, wherein the method comprises: - Determining the existence of a predetermined environmental situation, and - Detect that, despite the second deviation, no malfunction exists. Method according to any one of claims 1 to 4, wherein the determination of the first and / or the second deviation is carried out during a journey in the motor vehicle and / or after the journey by means of a data processing device external to the motor vehicle. Method according to claim 5, wherein data used by the automated driving function are stored in a ring buffer when the determination of the first and / or the second deviation is carried out during the journey. Method according to claim 6, wherein the data stored in the ring buffer are sent from the motor vehicle to the data processing device external to the motor vehicle when the malfunction is detected based on the first and / or the second deviation.Method according to any one of claims 1 to 7, wherein the malfunction is detected when the first and / or the second deviation exceeds a respective predetermined limit value. Device for data processing, comprising means for carrying out the method according to any one of claims 1 to 8. Computer program and / or computer-readable medium, each comprising instructions that are executed by a computer during the execution of the program or the instructions. Computers cause this to execute the method according to any one of claims 1 to 8.

Images