CREATING TEST DATA WITH CONSISTENT INITIAL CONDITIONS TO STIMULATE AN ECU UNDER TEST

Description

[0001] The invention relates to a method for processing raw data from the real world recorded by a sensor into test data for stimulating a control unit to be tested, comprising the following method steps: Providing recorded raw data from the real world, which has been recorded by means of a sensor for a recording duration along a recording path of at least a part of the surroundings of the recording path and has temporally successive data value sets that have resulted from real objects detected by the sensor, and determining the real objects detected by the sensor from the temporally successive data value sets and creating temporally successive path data sets, each describing a scene with images of these real objects at successive points in time.

[0002] In the fields of Advanced Driver Assistance Systems (ADAS) and Autonomous Driving (AD), the validation of sensor, sensor fusion, and perception components is an important and safety-critical undertaking. Due to the complexity and criticality of this validation, very high demands are placed on their safety and accuracy. To achieve this, recorded real-world data is often used—that is, data collected while driving a real motor vehicle in the real world using sensors attached to the vehicle. Data recorded in this way is characterized by a level of realism and fidelity that is unattainable with synthetic simulation data using state-of-the-art technology.The validation and testing of ADAS / AD perception and sensor fusion components through the replay of recorded data, also called data reprocessing, data re-simulation, data playback and data replay, is therefore an important aspect in the development of ADAS / AD systems.

[0003] Achieving high precision and realism is complex and demanding. The system under test must be fed with the recorded data in essentially the same way as it would be during a real test drive. Heterogeneous data streams, including sensor and network / bus data, must be synchronized during the data replay process, even if their packet / message sizes may differ. Furthermore, the sheer volume of streamed data is constantly increasing. A further challenge is the complex real-time adaptation of the recorded data to pass end-to-end encryption and security checks in the system under test (SUT). Furthermore, efficient scaling of the test to continuously feed millions of driven kilometers and orchestrate multiple test systems is crucial.

[0004] Different solutions for different data replay use cases are known from the state of the art, regardless of whether the SUT is physically available (hardware in the loop - HIL) or is still in the early software development phase (software in the loop - SIL) and whether it is a simple sensor component or a complex central computation platform for autonomous driving.

[0005] WO2020 / 264276 A1 deals with a method for generating virtual scenarios for testing vehicles based on data collected from a real vehicle, whereby data from real objects are used to generate simulated objects whose position and size can change over time.

[0006] Conventional simulation systems allow ECUs to be stimulated with synthetic raw sensor data. However, if the initial state of the scenario on which the raw data is generated is not plausible, the ECU under test or a component downstream of the ECU may be unable to process the situation, resulting in the test failing. This can be particularly the case if the simulation begins while the virtual test vehicle is traveling at full speed, or if objects in the environment suddenly appear, disappear, or jump from one location to another in an unnatural manner.

[0007] Based on this, it is the object of the invention to provide a method for generating such test data for a control unit to be tested, which does not have any implausibilities that would lead to an abort of a test of the control unit to be tested.

[0008] This problem is solved by the subject matter of the independent patent claims. Preferred developments are found in the subclaims.

[0009] According to the invention, a method is thus provided for processing raw data from the real world recorded by a sensor into test data for stimulating a control unit to be tested, which method comprises the following steps: Providing recorded raw data from the real world, which has been recorded by means of a sensor for a recording period along a recording path of at least a part of the surroundings of the recording path and has temporally successive data value sets that have resulted from real objects detected by the sensor, determining real objects detected by the sensor from the temporally successive data value sets and creating temporally successive path data sets, each describing a scene with images of these real objects at successive points in time, providing a library with synthetic objects, determining the absolute speed of the sensor along the recording path at time zero of the recording period and in the case,that the absolute speed of the sensor along the recording path is different from zero at time zero of the recording period, supplementing the temporally successive path data sets for an introductory period before the first path data set with temporally successive supplementary data sets, such that the sensor has an absolute speed of zero at time zero of the introductory period and the temporally successive supplementary data sets have synthetic objects that show a quasi-continuous temporal sequence of movements between immediately successive supplementary data sets, which quasi-continuously lead to the images of the real objects of the first path data set at time zero of the recording period, and generating the test data for stimulating a control unit to be tested by converting the supplementary data sets to such raw data,as they would have been recorded by the sensor during the introductory period in the real world if the sensor had detected the synthetic objects in the temporally successive supplementary data sets as real objects, and supplementing the recorded raw data before the first track data set at time zero of the recording period with the raw data obtained by converting the supplementary data sets.

[0010] To solve the above-mentioned problem, the invention proposes the automated creation of an "introduction scenario" that starts from a plausible initial state and seamlessly transitions to the initial state of an original scenario derived from the raw data recorded from the real world by the sensor. The invention is particularly useful for data replay when the recording of raw data begins mid-journey or when only a section of a recording is to be used. According to the invention, the recorded raw data is expanded by an introductory scenario with virtual scenarios, so that the result can then be converted back into a quasi-raw data format with a plausible initial state.

[0011] If the sensor moves along the recording path through its surroundings while recording the raw data, the relative motion of the objects detected by the sensor to the sensor in the real world is always mathematically continuous. This applies regardless of whether the detected objects themselves are moving in the environment or not. Since the recording of data from the real world is, of course, discrete, namely with temporally consecutive data sets, there are always discontinuities in the mathematical sense from data set to data set at a relative velocity other than zero between the sensor and a detected real object. Therefore, the term "quasi-continuous" is used in the context of the present invention to describe the fact that, as in the real world, the objects do not suddenly appear and disappear again. Furthermore, the objects do not jump from one location to another.Rather, when creating the supplementary data sets, the invention aims, as in the real world, at a movement of the objects relative to the sensor that is "quasi-continuous" in the sense that a human observer would perceive a continuous movement when viewing a visual representation of the supplementary data sets up to the first distance data set, as in a film with temporally successive individual images (frames).

[0012] It is therefore a key aspect of the present invention that, in the event that the sensor speed is different from zero in the first (earliest) distance data set, i.e., at time zero of the recording period, introductory data sets are inserted before the distance data sets, which lead to the first distance data set in a quasi-continuous manner. This applies both to a quasi-continuous relative movement of all objects relative to the sensor and to a starting situation of the sensor in the first supplementary data set, which should begin at a speed of zero.

[0013] The absolute speed of the sensor is therefore the speed of the sensor relative to objects in the sensor's surroundings that are not moving within the environment itself. If the recorded data from the real world is data collected using a sensor attached to a motor vehicle while driving along a road, then real objects that are not moving in the environment are included, e.g., the road itself and fixed structures in the environment, such as buildings or plants. Objects that are moving within the environment can be, for example, other road users, such as other motor vehicles or pedestrians. The term "absolute speed" therefore refers to the speed of the sensor, or in the previously described case, the speed of the motor vehicle carrying the sensor, relative to fixed structures in the environment.

[0014] Furthermore, it should also be noted that the term "image of a real object" is to be understood in a mathematical sense. It is therefore not necessarily a pictorial representation of a real object in the sense of a graphic representation. Rather, the images of the real objects are data, namely part of the route data, and thus result from a mathematical mapping rule dependent on the respective sensor from the data sets recorded by the sensor, which have resulted from real objects detected by the sensor. Nevertheless, it is of course possible that the images of the real objects are embodied graphically.

[0015] The term "distance data sets, each describing a scene with images of these real objects at successive points in time," means that the distance data sets contain information that, on the one hand, describes the type or properties of the real objects, such as their shape, and, on the other hand, also information that indicates the position of such a real object within the recorded scene. In principle, this information can be present in very different forms. According to a preferred development of the invention, the distance data sets, each describing a scene with images of these real objects at successive points in time, are formed from temporally successive frames, each containing a pictorial representation of the scene.The successive frames (individual images) thus essentially form a film of what the sensor captured as it moved along the recording path. The images of the real objects are displayed graphically. However, this graphic representation is only one option of the invention; a visual representation of scenes is not absolutely necessary.

[0016] The invention further relates to the use of test data obtained according to a method described above to stimulate a control unit under test. Within the scope of such use, the method preferably comprises the following steps: Testing a control unit with test data obtained according to one of the preceding claims, and if the control unit shows, upon being subjected to the test data, that it does not accept the test data as raw data recorded in the real world, repeating the method steps described in detail above of providing recorded raw data from the real world, determining real objects detected by the sensor from the temporally successive data value sets and creating temporally successive track data sets, each describing a scene with images of these real objects at successive points in time, providing a library with synthetic objects, supplementing the temporally successive track data sets and generating the test data for stimulating a control unit to be tested.In this context, it is particularly preferred that, when repeating these method steps, images of virtual objects are made more similar to the corresponding real objects. Additionally or alternatively, when repeating these method steps, images of virtual objects that were not previously used, for example, due to a supposedly too small size or assumed irrelevance of the corresponding real objects, are preferably inserted.

[0017] Furthermore, the invention relates to a non-volatile, computer-readable storage medium having instructions stored thereon which, when executed on a processor, effect a method as described above.

[0018] The invention will be explained in more detail below using a preferred embodiment with reference to the drawings.

[0019] The drawings show Fig. 1 schematically shows a flow diagram of a method for processing raw data from the real world recorded by a sensor into test data for stimulating a control unit to be tested according to a first embodiment of the invention, Fig. 2 schematically shows a flow diagram of a method for processing raw data from the real world recorded by a sensor into test data for stimulating a control unit to be tested according to a second embodiment of the invention, Fig. 3 a semantic description of an initial real traffic situation that corresponds to the target situation of the introduction scenario, Fig. 4 a semantic description of the structure of the road in the introductory scenario, Fig. 5 a semantic description of dynamic objects in the introductory scenario and Fig. 6 a semantic description of the entire introductory scenario.

[0020] Out of Fig. 1 A schematic flow diagram of a method for processing raw real-world data recorded by a sensor into test data for stimulating a control unit under test is shown. This control unit can, for example, be a control unit that performs tasks in the area of ​​"Advanced Driver Assistance Systems" (ADAS) and "Autonomous Driving" (AD) in a motor vehicle and, in doing so, intervenes in the steering and speed control of the vehicle.

[0021] This method comprises the following method steps: In method step S1, pre-recorded raw data from the real world are provided, which were recorded by means of a sensor for a specific recording duration along a specific recording route from a part of the surroundings of the recording route. The recording route was therefore previously driven by a motor vehicle that had a corresponding sensor in order to record at least part of the surroundings of the recording route. In this way, temporally successive data value sets were generated which resulted from real objects detected by the sensor. Of course, it is possible and also preferred within the scope of the invention for the motor vehicle with which the route was previously driven for recording to have more than one sensor for recording the surroundings of the recording route.

[0022] Subsequently, in process step S2, real objects detected by the sensor are determined from the temporally consecutive data sets, and temporally consecutive track data sets are created, each describing a scene with images of these real objects at consecutive points in time. These track data sets thus represent an image of the environment captured while driving along the recording track.

[0023] Then, in process step S3, a library of synthetic objects is provided. These synthetic objects are, in principle, also images of real objects, but they do not represent a direct representation of a respective real object. If, for example, a synthetic object is intended to represent another motor vehicle, the library of synthetic objects will contain only a finite number of such synthetic objects, each of which is intended to represent a motor vehicle. Typically, such synthetic objects each describe a motor vehicle class, such as "compact car," "station wagon," "sedan," "SUV," "small van," or "truck." The synthetic objects thus essentially represent simplified images of the real objects. The same applies to synthetic objects that represent static objects, such as buildings or plants, in the vicinity of the recording path.In particular, the synthetic objects can be stored as 3D objects, on the basis of which a graphics engine can render a two-dimensional view of a given object from any perspective in a data format corresponding to the recorded raw data.

[0024] Subsequently, in method step S4, the absolute speed of the sensor along the recording path at time zero of the recording period is determined. If the absolute speed of the sensor along the recording path at time zero of the recording period is different from zero, the temporally consecutive path data sets for an introductory period before the first path data set are supplemented with temporally consecutive supplementary data sets such that the sensor has an absolute speed of zero at time zero of the introductory period, and the temporally consecutive supplementary data sets contain synthetic objects that exhibit a quasi-continuous temporal movement sequence between immediately consecutive supplementary data sets, which quasi-continuously leads to the images of the real objects of the first path data set at time zero of the recording period.This is therefore the procedural step in the procedure described here in which a plausible introduction scenario is created in order to avoid implausible states that the control unit to be tested later would not accept.

[0025] Then, in method step S5, the test data for stimulating a control unit under test is generated by converting the supplementary data sets into raw data that would have been recorded by the sensor in the real world during the introductory period if the sensor had detected the synthetic objects in the chronologically consecutive supplementary data sets as real objects. In step S6, the recorded raw data before the first route data set at time zero of the recording period is supplemented with the raw data obtained by converting the supplementary data sets. In this way, implausible states can be avoided during control unit testing, which would otherwise lead to the abort of the control unit test.

[0026] This concludes the present method for processing the raw data from the real world recorded by the sensor into test data for stimulating the control unit under test. The test data generated in this way can be used multiple times and, moreover, for different control units under test. Furthermore, by traversing different recording paths, different sets of test data can be generated in order to test such control units. The invention thus makes it possible to generate different sets of test data, which can be stored in a separate library and used as needed to test control units. Nevertheless, it is possible, as schematically shown in the Fig. 2 The flow chart shown shows that after the test data has been created, the testing of a control unit can be carried out directly in a process step S7.

[0027] Such a procedure is explained below using the example of an introduction scenario for a radar raw data acquisition.

[0028] In Fig. 3A semantic description of the traffic situation is shown as it exists in the first route data set at time zero of the raw data recording period. Instead of the term "route data set," the term "frame" is also used below. This situation in the first route data set or in the first frame is the target situation, into which the introductory scenario should lead as seamlessly as possible. In addition to the radar sensor, other sensors can be used to support the reconstruction of the traffic scenario, such as a forward-facing camera that runs during the test drive. The semantic description of the target situation includes a label, a position, and a speed for each detected object.The introduction scenario includes the position of the real vehicle 1 with which the route data sets were determined, images of other passenger cars 2, images of trucks 3, images of vegetation 4, images of lane markings 5 ​​if the data do not originate exclusively from a radar sensor, and an image of a guardrail 6. For illustration purposes, . Fig. 3 also the cone of vision 7 of the radar sensor 8, which is carried by the vehicle 1 and with which the data was recorded.

[0029] To set up the introductory scenario, the first step is to create a virtual replica of a road section 10 that matches the road in the target situation, ie with the appropriate number of lanes and the appropriate width. This is shown schematically in Fig. 4visible. The length of the road section 10 should be selected such that sufficient space remains for all synthetic objects that are intended to lead to the images of the real objects of the first route data set at time zero of the recording duration, in particular for those that represent dynamic objects and move absolutely within the space of the scenario. A virtual test vehicle 9, also called an ego vehicle, is positioned on the road at the transverse position corresponding to the position of the real test vehicle 1 in the target scenario and then accelerated from zero to the target speed, i.e. the speed of the test vehicle in the target scenario (in the first route data set). The virtual test vehicle 9 is equipped with a virtual radar sensor 11 at exactly the location where the real radar sensor 8 was mounted in the test vehicle.

[0030] When the virtual test vehicle 9 is adjusted to the target speed, as shown schematically in Fig. 5 As shown, for each dynamic object from the target situation, a suitable virtual object that generates a similar radar image is positioned at the same relative position. Each dynamic object is then moved to a starting position by backward extrapolation, represented by arrows in the figure, of the respective dynamic object based on its absolute velocity in the target scenario. This extrapolation is carried out to such an extent that all starting positions are outside the cone of view 12 of the virtual radar sensor 11.

[0031] If the relative speed of a dynamic object is zero or nearly zero, this object is positioned just outside the cone of view 12 of the virtual radar sensor 11 and the relative target position of the object is specified as a setpoint to a controller that controls the object.

[0032] Now the scenes from Fig. 4 and Fig. 5 combined into a scenario, as in Fig. 6 until all dynamic objects are in the correct target position. In addition, Fig. 6It can be seen that, starting from the position where the virtual test vehicle 9 has been placed, the introductory scenario is now populated with static objects from the target scenario, so that in the last frames of the introductory scenario, at least the most important static objects are already visible, which are also visible in the first frame, i.e., in the first route data set corresponding to the target scenario, and no static objects appear out of nowhere. An example is shown of how, in the immediate vicinity of the target position of the virtual test vehicle 9, which it occupies in the last frame of the introductory scenario, the introductory scenario is populated with synthetic counterparts of the guardrail 6 and the vegetation 4.

[0033] The introduction scenario is then simulated. This simulation is converted into a synthetic radar raw data recording and then appended to the real recording so that the two merge seamlessly. This is followed by a test phase in which it is checked whether the control unit under test accepts the introduction scenario—that is, whether the transition from the introduction scenario to the real recording takes place without error messages or unusual behavior from the control unit.

[0034] If this is the case, the process is complete. If not, improvements are made. For example, virtual objects, both static and / or dynamic, are made more similar to their real counterparts from the recording, and / or additional static objects are inserted that were previously ignored due to their small radar cross-section. The basic principle here is that greater care is taken with objects that are of interest to the respective control unit. For example, a lane departure warning system algorithm will pay particular attention to lane boundaries such as shoulders, guardrails, curbs, etc., and tend to ignore other objects such as vehicles, pedestrians, and vegetation. Improvements are made until the respective control unit accepts the introduction scenario. List of reference symbols

[0035] Position of the real vehicle 1 Pictures of other passenger cars 2 Pictures of trucks 3 Pictures of vegetation 4 Images of lane markings 5 Image of a guardrail 6 radar sensor 7 Cone of view of the radar sensor 8 virtual test vehicle 9 section of road 10 virtual radar sensor 11 Cone of view of virtual radar sensor 12

Claims

1. A computer-implemented method for processing raw data recorded by a sensor from the real world into test data for stimulating a control unit to be tested, comprising the following method steps: providing recorded raw data from the real world recorded by means of a sensor for a recording period along a recording path of at least a part of the environment of the recording path, said data comprising temporally consecutive data value sets having resulted from real objects detected by the sensor, determining real objects detected by the sensor from the consecutive data value sets and creating consecutive path data sets, each of which describes a scene having images of said real objects at consecutive points in time, providing a library having synthetic objects, determining the absolute velocity of the sensor along the recording path at time zero of the recording period and, in the event that the absolute velocity of the sensor along the recording path at time zero of the recording period is different from zero, supplementing the temporally consecutive path data set for an initiation period before the first path data set having temporally consecutive supplemental data sets, such that the sensor has an absolute velocity of zero at time zero of the initiation period and the temporally consecutive supplementary data sets have synthetic objects showing a quasi-synchronous temporal sequence of movements between temporally immediately consecutive supplementary data sets quasi-synchronously leading to the images of the real objects of the first path data set at time zero of the recording period, and generating the test data for stimulating a control unit to be tested by converting the supplemental data sets to such raw data as would have been recorded by the sensor during the initiation period in the real world if the sensor had detected the synthetic objects in the successive supplemental data sets as real objects, and supplementing the recorded raw data before the first path data set at time zero of the recording period with the raw data obtained by converting the supplemental data sets.

2. The method according to claim 1, wherein the path data sets, each describing a scene having images of said real objects at successive points in time, are formed by successive frames, each containing a pictorial representation of the scene.

3. The method according to claim 1 or 2, wherein the synthetic objects showing a quasi-synchronous temporal sequence of movements between immediately successive supplementary data sets leading quasi-synchronously to the images of the real objects of the first path data set at time zero of the recording period, are selected in such a way that said objects are considered to be within a predetermined similarity range to the images of the real objects to which said objects lead in the context of an assessment by means of a predetermined metric or in the context of an assessment by a correspondingly trained artificial neural network.

4. The method according to claim 3, wherein a synthetic object is considered to be in the predetermined similarity range to the image of a real object when the raw data that would have been recorded by the sensor from a real object corresponding to the synthetic object is in a predetermined similarity range to the raw data recorded by the sensor from the real object to whose image the synthetic object leads.

5. The method according to any one of the preceding claims, wherein the sensor comprises a radar sensor, a lidar sensor, a camera, and / or an ultrasonic sensor.

6. The method according to any one of the preceding claims, having the following steps: testing a control unit by means of test data obtained according to any one of the preceding claims, and repeating the method steps of claim 1 if the control unit, when exposed to the test data, shows that said device does not accept the test data as raw data recorded in the real world.

7. The method according to claim 6, wherein, when repeating the method steps of claim 1, images of virtual objects are made more similar to the real objects corresponding thereto.

8. The method according to claim 6 or 7, wherein when repeating the method steps of claim 1, images of such virtual objects are inserted that have not been used before due to an insufficient size of the real objects corresponding thereto.

9. A use of test data obtained by a method according to any one of the preceding claims for stimulating a control unit to be tested.

10. A non-volatile, computer-readable storage medium having instructions stored thereon which, when executed on a processor, cause a method according to any one of claims 1 to 8 to be carried out.

11. A non-volatile, computer-readable storage medium having test data stored thereon which has been obtained according to any one of claims 1 to 8.