Method and apparatus for verifying the robustness of a sensor data evaluation using replay techniques

Abstract

The invention relates to a method for verifying the robustness of a technical system (1) with a control unit (2) and a plurality of sensor devices (3) that transmit sensor data to the control unit (2), characterized in that the method comprises the following steps: a) Acquiring (S1) sensor data over a predetermined period from real-world operation of the technical system (1) and a reference output of the control unit as a result of the input of the sensor data, wherein data elements of the sensor data are each stored with a timestamp or a temporal reference, b) Disconnecting the sensor devices (3) from the control unit (2) and feeding the recorded sensor data into the control unit (2) by means of a replay unit (5), c) Selectively varying (S2) the temporal assignment of the data elements of the recorded sensor data to each other, d) Determining an output (A) of the control unit (2) for each variation of the sensor data.e) Comparing the determined outputs (A) with the reference output (A_ref), f) Determining the robustness, depending on a proportion of the deviations between the determined outputs (A) and the reference output (A_ref).

Description

Technical field The invention relates to systems for evaluating and further processing sensor data from a plurality of sensor units, and in particular measures for checking the robustness of the evaluation system with regard to temporal offsets of the sensor data supplied by the sensor units or other disturbances in the supplied sensor data. Technical background Electronic control units (ECUs) typically evaluate sensor data from a multitude of sensors to perform specific predefined functions. For example, an ECU for a vehicle system executes functions and processes a variety of sensor data that can be supplied by numerous environmental sensors, such as cameras, LiDAR, radar, ultrasound, and the like. The sensor data is often combined within the ECU, for example, through sensor fusion, to obtain the best possible representation of the vehicle's surroundings. This representation is then further analyzed using additional functions to, for example, recognize environmental objects and derive situational descriptions of the vehicle's environment. The sensor devices typically include preprocessing capabilities for processing raw sensor data and transmit the acquired and, if necessary, preprocessed sensor data to the control unit according to a predefined transmission protocol. There, the sensor data is temporarily stored in sensor data buffers or other intermediate storage and read out and / or evaluated in its entirety at predetermined times. It is therefore necessary that the sensor data from each sensor device is available in the memory buffer as up-to-date as possible. Due to the data being transferred from the sensor data buffer at discrete times and the generally asynchronous provision of sensor data by the individual sensor devices, the recency of the relevant sensor data can vary depending on the assigned sensor device and the corresponding transmission channel. This means that the sensor data simultaneously transferred from the sensor data buffer to the control unit does not necessarily represent the system state detected by the sensor devices at a specific point in time. Therefore, sensor data representing a vehicle state, such as the current vehicle environment at a specific point in time, can exhibit time offsets. This can often be corrected through sensor fusion, the aggregation of sensor data, or by downstream functions within the control unit. However, if offsets between sensor data from different sensor devices increase, malfunctions can occur, which can be particularly safety-critical when using the aggregated sensor data for autonomous driving functions. It is therefore necessary to verify the robustness of a system design, especially with regard to time offsets or other errors that can occur when using sensor data from different sensor devices. Disclosure of the invention This problem is solved by the method for robustness testing of a technical system with a control unit and a plurality of sensor devices using Replay according to claim 1 and by a corresponding device according to the dependent claim. Further details are specified in the dependent claims. According to a first aspect, a procedure is provided for checking the robustness of a technical system with a control unit and a plurality of sensor devices that transmit sensor data to the control unit.The procedure comprises the following steps: a) Acquiring sensor data over a predetermined period from real-world operation of the technical system and a reference output from the control unit as a result of the sensor data input, whereby data elements of the sensor data are each stored with a timestamp or a temporal reference, b) Disconnecting the sensor devices from the control unit and feeding the recorded sensor data into the control unit using a replay unit, c) Targeted variation of the temporal assignment of the data elements of the recorded sensor data to each other, d) Determining an output from the control unit for each variation of the sensor data, e) Comparing the determined outputs with the reference output, f) Determining the robustness, depending on the result of the comparisons between the determined outputs and the reference output. Technical systems often include a control unit that continuously receives sensor data from a variety of sensors. This sensor data is received and processed by the control unit's microcontroller or microprocessor and further processed based on the intended functions. Each sensor may have its own logic for preprocessing the raw sensor signals and transmit the preprocessed sensor data to the control unit. The transmission of sensor data, which is acquired almost continuously, can also occur in packet-based form or according to specific transmission protocols. This can lead to the received sensor data from the sensor devices being recorded with a time delay. Therefore, it is not guaranteed that the most recent received sensor data always reflects the system state at a given time. Different times, however, represent different system states. For example, camera data is often transmitted in frame rates of 30 to 50 ms, while sensor devices such as lidar, radar, and the like can have a frame rate of 10 ms to 60 ms and are often not synchronized with the cameras. The sensor data receiver in the control unit typically writes the received sensor data to a sensor data buffer. This data is then read out or transferred to a microprocessor or microcontroller for further processing at predetermined times, which may be asynchronous with the transmission cycles and acquisition frequencies. Therefore, the sensor data buffer in the control unit can contain data from different system states at different times for temporary storage. Depending on the state of the technical system, environmental conditions, and other factors, time delays in the system states detected by the sensors can be stored in the sensor data buffer, meaning that the evaluation is not based on simultaneously detected states. This is generally tolerable within certain limits, as the time interval for evaluating the system states is often sufficiently short. However, the time delays between the sensor data from the sensors are not deterministically predictable. In extreme cases, time delays can also occur between the system states detected by the sensors, which can lead to errors during the subsequent aggregation of the sensor data or during the evaluation of the aggregated sensor data in the control unit.However, particularly for safety-critical functions, such as those used in autonomous driving, a high degree of robustness is required with regard to different time offsets between sensor data or other errors. Furthermore, errors in the transmission of sensor data to the control unit can lead to data loss or multiple transmissions, which can also result in errors during evaluation in the control unit. Another problem is that the processing of sensor data in the control unit is not deterministic. This means that even if sensor data is provided to the control unit at precisely the same time, and the control unit has an identical or predefined initial state, the internal execution of the process steps for realizing the control unit's function will differ. Depending on the timing of the sensor data, this can lead to permanent or only intermittent errors in the execution of the control unit's function. This necessitates verifying the robustness of the technical system, i.e., checking how robustly the technical system maintains its function even with non-deterministic sensor data processing behavior. The non-deterministic behavior of sensor data processing in the control unit is primarily due to the processing of sensor data in parallel processors, the use of DMA access, shared resources requiring sequential processing or nesting, the use of data caches, and memory properties such as DDR Refresh and Precharge. This means that if the system is repeatedly stimulated with precisely the same sensor data sequences and the same controllable preconditions—that is, all sensor devices deliver exactly the same information at the same time, and the system is in exactly the same state at the start—the internal execution at intermediate steps will never be identical. Typically, to test technical systems, sensor data from a current operational scenario in the field is recorded and made available for verification at the control unit's interface. The output of the control unit function is also captured with a timestamp and stored as a reference output along with the sensor data. This allows, for example, the evaluation of the control unit's sensitivity to deviations in the internal data processing due to the non-deterministic behavior of the microprocessor by comparing the control unit's output during sensor data testing with the reference output. Thus, while an error that has occurred can be reconstructed and potentially non-deterministic sporadic errors can be found, no statement can be made about the robustness of the technical system against jitter, varying time offsets between data elements of the sensor data. To test a technical system for robustness with regard to inconsistency, time offsets between data elements, and data gaps in sensor data from multiple sensor devices, a replay system is proposed. This system continuously monitors sensor data transmitted to the control unit via corresponding data lines during test bench measurements, field measurements, or measurements during operation, and records the transmitted data streams in their temporal relationship to one another. In other words, a record is created of the sensor data streams transmitted by the sensor devices, with the individual data packets or elements of the sensor data streams having the same temporal relationship to each other as they were transmitted during acquisition.In particular, each data element is assigned a time reference or timestamp, so that the time of its arrival at the control unit is precisely defined in relation to the other data elements or data packets. Similarly, the control unit's reference output is recorded with corresponding time references. Replay is the process of feeding previously recorded or artificially generated data into the system by replacing / emulating the original data source. The general goal of replay is to make unique processes or scenarios repeatable for analysis, optimization, debugging, verification, or validation purposes. There are two fundamentally different types of replay based on the dependency between injected and measured data, i.e., the feedback behavior. In open-loop (XoL) applications, the injected data includes previously recorded data. There is no feedback loop between injected and measured data during the replay process. This type is primarily used for troubleshooting or verification. In closed-loop (XiL) applications, the injected data is generated in real time from the measured data to also capture the system's response to the injected data in real time. The non-deterministic behavior of data processing is not inherently critical, as long as the final result or response meets expectations and requirements. Therefore, it is the responsibility of the ECU developers to design the system to be robust enough that the result is acceptable in every single case. One approach to robustness testing of the technical system involves first acquiring sensor data from regular operation (field or test bench operation) of the system, including the ECU output as a reference. The data is acquired in such a way that, using the sensor data's timestamps, it is possible to reproduce the sensor data in the exact sequence and temporal relationship to one another, ensuring that the execution of the ECU function based on the sensor data can always be performed according to the same scenario. The sensor data from multiple sensor devices is typically stored in channels, meaning it is transmitted via different data paths or distributed as data packets assigned to different sensor devices. It is now possible to modify the timing of sensor data assigned to a specific sensor device relative to the other sensor data by selectively delaying, advancing, or omitting a portion of that sensor's data. This allows for the simulation of various possible data transmission scenarios from that sensor device. Delaying or advancing data can be achieved by adjusting the respective timestamp or time reference. For robustness testing, data elements (data packets or data segments) of the sensor data of one or more of the sensor devices are shifted in time or omitted, whereby the time offset of the relevant data element of the sensor data is set within a predetermined time range with reference to the other sensor data of the sensor devices and the result of the execution of the control unit function is evaluated with regard to the original result or the reference output. It may be provided that the results of the comparisons indicate an error case if, accordingly, a measure of deviation between an output of the control unit and a reference output exceeds a predetermined threshold, whereby the robustness is determined by determining the ratio between the number of error cases and the number of all variations of the temporal assignments of the data elements performed, whereby an exceedance of a predetermined error threshold by the ratio serves as an indicator of insufficient robustness. If, as a result of the comparison, an output from the control unit deviates from the regular reference output with regard to the expected behavior, this can be considered an error. A deviation can occur if the output of the control unit differs from the regular reference output, or if the output of the control unit deviates from the regular reference output by more than a certain threshold, or if there is an unacceptable change in behavior. After numerous measurements in which the temporal properties of data elements from the sensor devices are varied, the ratio between faulty and fault-free control unit outputs can now be used to assess robustness. In particular, the technical system can be rejected or improved if the robustness value exceeds a predefined threshold. Brief description of the drawings The embodiments are explained in more detail below with reference to the accompanying drawings. These show: Fig. 1 a schematic representation of a technical system with a control unit and a plurality of sensor devices that supply sensor data to the control unit; Fig. 2 a flowchart illustrating a method for testing the robustness of the technical system with respect to sensor data variations. Description of embodiments Fig. 1 schematically shows a technical system 1 with a control unit 2 to which a plurality of sensor devices 3 are connected. The control unit 2 can have a microprocessor and / or a microcontroller that processes sensor data transmitted from the sensor devices 3 to the control unit 2, summarizes this data, and executes a subsequent control unit function. This merging can be equivalent to sensor fusion. For example, camera image data, lidar data, and radar data can be fused to obtain a representation of a vehicle's surroundings. The control unit function results in an output that can be used in downstream systems or devices to perform actions and control vehicle actuators. An example of the technical system could be a control unit for a motor vehicle that manages the vehicle's operation in autonomous or semi-autonomous driving mode. The sensor devices 3 can include environmental sensors, such as one or more cameras, one or more radar sensors, a LiDAR system, and / or an ultrasonic system. Additional sensors can detect environmental conditions, such as temperature, humidity, wind speed, and the like. The control unit 2 processes the sensor data provided by each of the sensor devices 3. Data transmission between the sensor devices 3 and the control unit 2 can occur as packets or as a data stream, with each sensor device 3 potentially having preprocessing capabilities for raw sensor data that appropriately prepares the sensor data. The sensor data can be transmitted to the control unit 2 via different data paths or via a common data path, with the data packets being assigned to the respective sensor device 3 in the case of packet-based data transmission. Overall, the data transmission is structured so that each data element can be assigned to a specific sensor device 3. Due to differing latencies in the preprocessing of the raw sensor data and / or the transmission channel, e.g., due to data transmission interference, time delays and offsets can occur between data elements assigned to the different sensor devices. Data elements can also be lost and / or arrive at control unit 2 with a significant delay due to the need for retransmission. For example, the transmission rate of a camera image frame is approximately 30 ms, while frames from a LiDAR or radar can be transmitted at 10 ms. In the control unit 2, the sensor data from the various sensor devices 3 are read into an input data buffer and stored there in a manner assigned to the respective sensor device 3 and with a temporal reference to each other or with a timestamp. The input data buffers are read out by a microprocessor and / or microcontroller at a specific time, assuming that the sensor data stored in the input data buffers represent a system state at a specific time, and in particular that the sensor data of each of the sensor devices 3 represent a state at the same time. Data processing in the microprocessor and / or microcontroller is based on the assumption that the sensor data from the multiple sensor devices 3 were acquired at the same specific time. However, if the sensor data from the multiple sensor devices 3 represent different times, this can lead to artifacts, for example, during sensor fusion of camera, LiDAR, radar, and ultrasound data. These artifacts can distort the result of an evaluation of the environmental situation in a vehicle. Functions based on this, such as collision warning, object detection, and the like, can then produce erroneous outputs. This is safety-critical, and there is a need to ensure the robustness of a technical system 1. Figure 2 shows a flowchart illustrating the procedure for testing the robustness of the technical system 1. To test the robustness of the technical system 1, a replay unit 5 is therefore provided, which in step S1 during an ongoing operation of the technical system 1, for example on a test bench or in the field, records the sensor data for a specified period of time and also records the corresponding control unit outputs A. In step S2, the robustness can now be tested by varying the sensor data, in particular according to a predefined time shift of data elements of the individual sensor data, so that their relative relationship to each other is changed. For this purpose, the sensor devices 3 are disconnected from the control unit 2 and instead the previously recorded sensor data is sent to the control unit using the replay system 5. The order and temporal relationships of the data elements can be varied. Furthermore, individual data elements can be deleted, or the temporal order of data elements from a specific sensor device can be reversed. The data elements are now varied in arbitrary variations, defined accordingly within a range of maximum and minimum time shifts, and transmitted to control unit 2. The resulting control unit output A is compared in step S3 with the original regular reference output of control unit 2 using the original sensor data. If a deviation (difference, Euclidean distance of an output vector, etc.) between an output of the control unit (A) and a reference output (A_ref) exceeds a predefined threshold or defines acceptable behavior of the technical system 1, then an error has occurred. If the control unit output A essentially corresponds to the regular or an expected control unit output, then no error has occurred. Robustness can be indicated, for example, by the ratio of failures to the total number of test runs or variations. If the ratio exceeds a predefined error threshold, a lack of robustness can be signaled or displayed.

Claims

A method for verifying the robustness of a technical system (1) with a control unit (2) and a plurality of sensor devices (3) that transmit sensor data to the control unit (2), characterized in that the method comprises the following steps: a) Acquiring (S1) sensor data over a predetermined period from real-world operation of the technical system (1) and a reference output of the control unit as a result of supplying the sensor data to the technical system (1), wherein data elements of the sensor data are each stored with a timestamp or a temporal reference, b) Disconnecting the sensor devices (3) from the control unit (2) and feeding the recorded sensor data into the control unit (2) by means of a replay unit (5), c) Targeted variation (S2) of the temporal assignment of the data elements of the recorded sensor data to each other using the replay unit (5).d) Determining an output (A) of the control unit (2) for each variation of the sensor data, e) Comparing the determined outputs (A) with the reference output (A_ref), f) Determining the robustness, depending on a result of the comparisons between the determined outputs (A) and the reference output (A_ref). Method according to claim 1, characterized in that the targeted variation (S2) of the temporal assignment of the data elements of the recorded sensor data to each other comprises that at least a part of the data elements of the sensor data of one or more sensor devices (3) is shifted, brought forward, delayed, omitted or exchanged in their order within a predetermined time range. Method according to one of the preceding claims, characterized in that when varying the temporal assignment, selected data elements of at least one sensor device (3) are completely or partially deleted or provided multiple times in order to evaluate the effect of transmission errors or data loss. Method according to one of the preceding claims, characterized in that each data element is assigned a timestamp, wherein the timestamp of at least one of the data elements is modified, wherein the data elements are fed in in temporal relation to each other according to the timestamps. Method according to one of the preceding claims, characterized in that the temporal assignment is varied by an algorithm for the systematic or stochastic modification of the timestamps of the recorded data elements. Method according to one of the preceding claims, characterized in that the results of the comparisons indicate an error case if, accordingly, a measure of deviation between an output of the control unit and a reference output exceeds a predetermined threshold, wherein the robustness is determined by determining the ratio between the number of error cases and the number of all variations of the temporal assignments of the data elements performed, wherein an exceedance of a predetermined error threshold by the ratio serves as an indicator of insufficient robustness. Device (replay unit) (5) for checking the robustness of a technical system (1), characterized by: - ​​at least one storage device for storing sensor data acquired during real operation of the technical system (1) together with timestamps and a reference output, - an interface for disconnecting the sensor devices (3) from the control unit (2) and for feeding modified sensor data into the control unit (2), - means for selectively varying the temporal assignment of data elements of the sensor data acquired during real operation, - an evaluation unit configured to compare the outputs (A) determined by the control unit (2) with at least one reference output (A_ref) and to indicate a robustness of the technical system (1) depending on the comparison results. Technical system (1) with a control unit (2) and a plurality of sensor devices (3), characterized in that it comprises a device according to one of claims 6 to 7, wherein the technical system (1) is configured to carry out a method according to one of claims 1 to 5. Computer program comprising commands which, when the computer program is executed in a device according to claim 6 or 7, cause the device to perform the steps mentioned therein. Machine-readable storage medium on which a computer program according to claim 9 is stored.

Images