Method for evaluating sensor model, method for training recognition algorithm, and sensor system

Abstract

The invention relates to a method for evaluating a sensor model, wherein real sensor data is compared with simulated sensor data. To this end, the following steps are carried out. First, sensor parameters to be analyzed are specified and a value range is specified for each of the sensor parameters to be analyzed. Subsequently, input values are generated for the sensor model, a first input value being generated by simulating the behavior of the sensor, and a second input value comprising a real sensor value. A simulation of the behavior of the sensor is then assessed with respect to the accuracy and with respect to the calculation time. Finally, a sensitivity analysis is performed by means of the accuracy and the calculation time.

Description

Technical Field

[0001] The invention relates to a method for evaluating a sensor model, a method for training a recognition algorithm, and a sensor system. Background Art

[0002] For example, sensor systems used in vehicles, especially motor vehicles, are known. In this regard, the focus of current developments is on the automatic execution of driving functions, starting from driver assistance systems that support the driver of the vehicle to vehicles that can be driven in a certain area or even completely autonomously. In order to automatically execute the driving function, it is necessary to provide a sensor system in the vehicle, with which the environment of the vehicle can be identified and classified, so that this information can be used in trajectory planning and / or the driving function can be controlled based on this information. The sensor system, including the evaluation electronics, must be tested and verified in a costly manner before being used in the vehicle, in order to determine as much as possible that all relevant information is identified by the sensor system and that relevant information is not ignored. This can be done in particular with the help of simulated sensor data, which has the advantage that relevant situations can be clearly inserted into the simulated sensor data, and the sensor system can thus be evaluated in view of all relevant situations. If only real sensor data are used, this can only be achieved through extremely extensive test drives even in an ideal situation, wherein even this cannot guarantee that all possible situations are actually covered.

[0003] The relevant sensor models are then created for the sensor system. Based on these sensor models, the behavior of the sensor system in the simulated environment can be evaluated. Using the simulated environment and the behavior of the sensor system tracked therein, other elements of the control can be evaluated, such as object recognition, wherein the number of test drives can be reduced. However, these simulations are very complex and require high computing performance. Summary of the invention

[0004] The object of the present invention is to provide a method for evaluating a sensor model with reduced required computing performance. Another object of the present invention is to provide a method for training a recognition algorithm, which is based on the method according to the present invention for evaluating a sensor model. Another object of the present invention is to provide a sensor system, which is trained using the method according to the present invention for training a recognition algorithm. These objects are achieved by the subject matter of the independent patent claims. Advantageous improvements are described in the dependent patent claims.

[0005] In a method for evaluating a sensor model, real sensor data are compared with simulated sensor data. To this end, the following steps are performed. First, the sensor parameters to be analyzed and, respectively, a value range for each of the sensor parameters to be analyzed are specified. Subsequently, input values ​​are generated for the sensor model, wherein the first input value is generated with the aid of a simulated sensor behavior and the second input value comprises the real sensor value. The simulation of the sensor behavior is then evaluated with respect to accuracy and with respect to computing time. Finally, a sensitivity analysis is performed with the aid of the accuracy and computing time.

[0006] By means of the sensitivity analysis, those sensor parameters which require excessively much computing time compared to the accuracy gain of the sensor parameters or their value ranges can be adapted. This enables better utilization of existing computing power and thus provides an improved sensor model, in which less computing performance is required compared to the accuracy. Thus, a more efficient sensor model can be provided.

[0007] The second input value can be directly the real sensor value. In addition, the second input value can also be a value derived from the real sensor value. Both should be included by the expression "comprising".

[0008] If necessary, the sensor parameters can be different for the sensor system to be simulated. Radar sensor systems, lidar sensor systems and camera sensor systems may be particularly relevant. For radar sensor systems, the sensor parameters can include in particular the standard deviation of the average noise of the sensor data, a shift in the probability of detection, the maximum antenna gain, a measure of the noise, overall system losses and / or an average radar cross section. For lidar sensor systems, the sensor parameters can include in particular the number of scanning layers, the arrangement of the scanning layers, the intensity, motion blur, rolling shutter and / or atmospheric attenuation. In the case of a camera sensor system, the sensor parameters can include in particular vignetting, lens distortion, the influence of color filters, a windshield model and / or weather influences.

[0009] In one embodiment, the quotient of accuracy and computing time is determined during the sensitivity analysis. Subsequently, that sensor parameter of the sensor parameters for which the quotient of accuracy and computing time is as small as possible is set to a specified value. This makes it possible to simplify the sensor model, wherein that sensor parameter is set to a specified value, wherein the ratio of accuracy to required computing time is as favorable as possible. For example, a first sensor parameter may have a greater influence on the simulation result than a second sensor parameter. However, if more computing time can be saved by setting the first sensor parameter to a specified value compared to a poorer accuracy, it may nevertheless be advantageous to ignore the sensor parameter in the simulation by setting the sensor parameter to the specified value.

[0010] In one embodiment, the method is repeated. During the repetition, the sensor parameter of the other sensor parameters for which the quotient of accuracy and calculation time is as small as possible is set to a specified value. This allows a further simplification of the sensor model.

[0011] In one specific embodiment, the method is repeated until all sensor parameters are set to specified values ​​and the quotient of accuracy and computation time for all sensor parameters is below a predefined value. This allows a further simplification of the sensor model.

[0012] In one specific embodiment, a comparison is made between the first input value and the second input value in order to assess the simulation of the behavior of the sensor with respect to the accuracy. This can be used, in particular, to determine to what extent the simulation of the behavior of the sensor corresponds to the real sensor data. For example, during the simulation, a real drive of the vehicle can be re-adjusted with the corresponding sensor system and the real sensor data can then be compared with the simulated sensor data. This is a method that is simple to implement.

[0013] In one specific embodiment, the calculation time is measured when evaluating the simulation of the behavior of the sensor with respect to the calculation time. In particular, the calculation time can be measured during a time when the sensor parameter is set to a defined value and a time when the sensor parameter is variable. The difference between these two calculation times then results in the calculation time, based on which the simulation of the behavior of the sensor is evaluated with respect to the calculation time.

[0014] In one specific embodiment, the second input value is generated by means of a sensor.

[0015] In one specific embodiment, a scenario for the first input value is defined as a function of the second input value. This can allow the predefined scenario that is readjusted using the real measurement drive to be changed so that scenarios for which no real measurement drive was performed can now also be considered.

[0016] The method according to the invention for evaluating a sensor model can be used in a method for training a recognition algorithm. In the method for training a recognition algorithm, the recognition algorithm is trained using simulated sensor data, wherein the simulated sensor data are evaluated using the method according to the invention for evaluating a sensor model. As a result, simulated sensor data can be generated such that, compared to the methods known in the prior art, an improved utilization of the available computing time can be achieved during the simulation.

[0017] The invention further comprises a sensor system, in particular a sensor system for use in a vehicle, which has a sensor and an evaluation unit. The evaluation unit is configured to execute a recognition algorithm. The recognition algorithm is trained using the method according to the invention for training a recognition algorithm. The sensor system can be, in particular, a radar sensor system with a radar sensor, a lidar sensor system with a lidar sensor and / or a camera sensor system with a camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Embodiments of the present invention are explained with reference to the following drawings. In the schematic diagram:

[0019] Figure 1 showing a vehicle;

[0020] Figure 2 A computing unit is shown;

[0021] Figure 3 A first flow chart showing a method for evaluating a sensor model;

[0022] Figure 4 A second flow chart showing a method for evaluating a sensor model;

[0023] Figure 5 a graph showing the accuracy ratings; and

[0024] Figure 6 Graph showing computation time ratings. DETAILED DESCRIPTION

[0025] Figure 1A vehicle 10 is shown with a sensor system 100. The sensor system 100 comprises at least one sensor 110 and an evaluation unit 120. The sensor system 100 may comprise a radar sensor system 100 with a radar sensor 110, a lidar sensor system 100 with a lidar sensor 110 and / or a camera sensor system 100 with a camera 110. Therefore, more than one type of sensor system 100 may also be arranged in the vehicle 10. The evaluation unit 120 is hereby respectively configured to process the signals of the sensor 110 and output the signals to further control units 11, 12. The further control units 11, 12 may hereby comprise a control unit 11 for automatically performing a driving function or a control unit 12 for further functions. The control unit 11 for automatically performing a driving function may hereby be assigned to a driver assistance system that assists the driver of the vehicle 10. Alternatively, it may be provided that autonomous driving in certain areas or completely autonomous driving may be possible with the aid of the control unit 11 for automatically performing a driving function. In order to automatically perform driving functions, it is necessary to provide a sensor system 100 in the vehicle 10, with which the environment of the vehicle 10 can be detected and classified in order to use this information in trajectory planning and / or to control driving functions as a function of this information. The sensor system 10, including the evaluation unit 120, must be tested and verified in a complex manner before use in the vehicle 10 in order to ensure as far as possible that all relevant information is detected by the sensor system 10 and that no relevant information is ignored. Figure 1 The vehicle shown in FIG. 1 can be used in a method for evaluating a sensor model and in particular provide real sensor data, wherein the real sensor data can be recorded as physical measurement data by means of sensor 110 and can be converted into digital data by means of evaluation unit 120 .

[0026] Figure 2 A calculation unit 200 is shown, with which a method for evaluating a sensor model can be carried out. The calculation unit 200 has a processor 210, wherein a computer program can be run on the processor 210. In addition, the calculation unit 200 has an input interface 220 and an output interface 230. For example, real sensor data determined by the sensor system 100 of the vehicle 10 can be read in via the input interface 220. Simulation results can be output via the output interface 230.

[0027] Computer unit 200 can also be configured to generate simulated sensor data. In this case, relevant situations can be explicitly inserted into the simulated sensor data, and sensor system 100 can thus be evaluated with regard to all relevant situations. If only real sensor data were to be used, this would only be possible, even in an ideal case, through extremely extensive test drives, wherein even this does not guarantee that all possible situations are actually covered.

[0028] Figure 3 A flow chart 300 is shown of a method for evaluating a sensor model, wherein real sensor data are compared with simulated sensor data. The real sensor data may be, in particular, Figure 1 As explained above, the sensor system 100 can be used to record the Figure 2 The method is executed on a computer unit 200 as shown in FIG.

[0029] In a first method step 301, the sensor parameters to be analyzed are specified and the value ranges for each of the sensor parameters to be analyzed are determined. In a second method step 302, input values ​​for the sensor model are generated, wherein the first input value is generated by means of a simulated behavior of the sensor 110 and the second input value comprises a real sensor value, in particular a real sensor value of the sensor 110. In a third method step 303, the simulation of the behavior of the sensor 110 is evaluated with respect to accuracy. In a fourth method step 304, the simulation of the behavior of the sensor 110 is evaluated with respect to computing time. In a fifth method step 305, a sensitivity analysis is performed with respect to accuracy and computing time. The third method step 303 and the fourth method step 304 can also be performed in reverse order or simultaneously.

[0030] In one embodiment, the quotient of accuracy and computation time is determined during the sensitivity analysis in fifth method step 305. Subsequently, the sensor parameter for which the quotient of accuracy and computation time is as small as possible is set to a specified value. This allows the sensor model to be simplified, wherein computing unit 200 requires less computing power to simulate the simplified sensor model.

[0031] In one embodiment, the method is then repeated with all five method steps 301, 302, 303, 304, 305. The sensor parameter of the other sensor parameters for which the quotient of accuracy and computation time is as small as possible is now set to a defined value. This allows the sensor model to be further simplified.

[0032] In one exemplary embodiment, the method is repeated until all sensor parameters are set to defined values, at which the quotient of accuracy and computing time lies below a predefined value.

[0033] Figure 4 A flow chart 300 is shown of a method for evaluating a sensor model, wherein a method of evaluating a sensor model is also performed in conjunction with Figure 3 The method steps 301, 302, 303, 304, 305 are explained and other options are described. If necessary, other options can also be set separately and supplemented. Figure 3In the parameter generation 307, sensor parameter values ​​are respectively generated within the value range of the sensor parameter to be analyzed. In the second method step 302, a real value generation 310 is performed by means of a test drive 311 of the vehicle 10, wherein during the test drive 311, sensor data 312 are generated by means of the sensor 110 or the sensor system 100. In addition, in the second method step 302, a synthetic generation 320 is performed, wherein a simulation 321 of the behavior of the sensor 110 or the sensor system 100 is performed, and sensor model data 322 are subsequently generated. The sensor model data 322 are here the first input values, and the sensor data 312 are the second input values. The sensor data 312 and the sensor model data 322 are then further processed in a test environment 325, and the third method step 303 and the fourth method step 304 are subsequently performed, and the test environment 325 can, for example, include the system to be tested (system under test). Finally, the fifth method step 305 is also performed in this method.

[0034] In one exemplary embodiment, a comparison is performed between the first input value and the second input value, ie, if necessary, sensor data 312 and sensor model data 322 , in order to assess the simulation of the behavior of sensor 110 with respect to the accuracy of the comparison.

[0035] In one exemplary embodiment, the computation time is measured in fourth method step 304 in order to evaluate the simulation of the behavior of sensor 110 with respect to the computation time. This can be done, for example, with the aid of a clock provided in computing unit 200 .

[0036] In one embodiment, the second input value is generated by means of the sensor 110. In one embodiment, a scenario for the first input value is defined based on the second input value. Figure 4 This is indicated by an arrow from the test drive 311 to the simulation 321 .

[0037] Figure 5 An accuracy diagram 400 is shown, wherein the accuracy 410 of a first sensor parameter 401, a second sensor parameter 402, a third sensor parameter 403, a fourth sensor parameter 404, and a fifth sensor parameter 405 are plotted. For each sensor parameter 401, 402, 403, 404, 405, the accuracy 410 is illustrated in the form of a bar graph, wherein the shaded area of ​​the bar graph indicates which accuracy would be lost if the corresponding sensor parameter 401, 402, 403, 404, 405 were set to a predetermined value, respectively. Thus, the smaller the shaded area, the smaller the effect that setting the sensor parameter 401, 402, 403, 404, 405 would have on the accuracy 410.

[0038] Figure 6A calculation time diagram 420 is shown, in which calculation times 430 are plotted for a first sensor parameter 401, a second sensor parameter 402, a third sensor parameter 403, a fourth sensor parameter 404 and a fifth sensor parameter 405. For each sensor parameter 401, 402, 403, 404, 405, the calculation time 430 is illustrated in the form of a bar code, wherein the shaded area of ​​the bar diagram indicates, respectively, which calculation time is possible if the corresponding sensor parameter 401, 402, 403, 404, 405 is set to a predetermined value. Therefore, the larger the shaded area, the more calculation time is possible by setting the sensor parameter 401, 402, 403, 404, 405.

[0039] for Figure 5 as well as Figure 6 , different numbers of sensor parameters 401, 402, 403, 404, 405 can also be set. Figure 5 and 6 The illustration of 401, 402, 403, 404, 405 makes it clear that individual sensor parameters, for example second sensor parameter 402, make only a small contribution to accuracy 410, but nevertheless cause a high calculation time 430. This can be used to set second sensor parameter 402 to a predetermined value within the value range of second sensor parameter 402, since calculation time 430 is thereby very significantly reduced and accuracy 410 is only insignificantly impaired. This can be expressed, for example, by the quotient of accuracy 410 and calculation time 430.

[0040] If the sensor 110 is a radar sensor or the sensor system 100 is a radar sensor system, the sensor parameters 401, 402, 403, 404, 405 can include in particular the standard deviation of the average noise of the sensor data, the shift of the detection probability, the maximum antenna gain, the measure of the noise, the overall system loss and / or the average radar cross section. The standard deviation of the average noise of the sensor data can be between 0 and 8 dB. The shift of the detection probability can be between -5 and 5. The maximum antenna gain can be between 10 and 25 dB. The measure of noise can be between 10 and 20. The overall system loss can be between 0 and 20 dB. The average radar cross section can be between -10 and 10 dBsm. If the sensor system 100 is a lidar sensor system and the sensor 110 is a lidar sensor, the sensor parameters can include in particular the number of scan layers, the arrangement of the scan layers, the intensity, motion blur, rolling shutter and / or atmospheric attenuation. If sensor system 100 is a camera sensor system and sensor 110 is a camera, the sensor parameters may include, among other things, vignetting, lens distortion, effects of color filters, windshield models and / or weather effects.

[0041] The invention further comprises a method for training a recognition algorithm, wherein the recognition algorithm is trained using simulated sensor data, wherein the recognition algorithm is trained using a combination of Figures 1 to 6 The explained method for evaluating a sensor model is used to evaluate simulated sensor data.

[0042] The present invention further comprises a sensor system 100 having a sensor 110 and an evaluation unit 120 , wherein the evaluation unit 120 is configured to execute a recognition algorithm, wherein the recognition algorithm is trained using the method according to the present invention for training a recognition algorithm.

[0043] Although the present invention has been described in detail through preferred embodiments, the present invention is not limited to the disclosed examples, and other variations may be derived therefrom by those skilled in the art without departing from the scope of protection of the present invention.

Claims

1. A method for evaluating a sensor model, wherein real sensor data are compared with simulated sensor data, the method comprising the following steps: - specifying sensor parameters to be analyzed and, respectively, specifying a value range for each of said sensor parameters to be analyzed; - generating input values ​​for the sensor model, wherein a first input value is generated by means of a simulated sensor (110) behavior and a second input value comprises a real sensor value; - a simulation of the behavior of the sensor (110) for evaluation with respect to accuracy (410); - a simulation of the behavior of the sensor (110) evaluated with respect to a computing time (430); - performing a sensitivity analysis with the aid of said accuracy (410) and said calculation time (430).

2. The method according to claim 1, wherein the quotient of accuracy (410) and calculation time (430) is determined during the sensitivity analysis, and the sensor parameter for which the quotient of accuracy (410) and calculation time (430) is as small as possible is set to a specified value.

3. The method according to claim 2, wherein the method is repeated and wherein that sensor parameter of the other sensor parameters for which the quotient of accuracy (410) and calculation time (410) is as small as possible is set to a specified value.

4. The method according to claim 2 or 3, wherein the method is repeated until all sensor parameters are set to specified values, and in the case of all sensor parameters, the quotient of accuracy (410) and calculation time (430) is below a predetermined value.

5. The method according to claims 1 to 4, wherein a comparison is performed between the first input value and the second input value for assessing the simulation of the behavior of the sensor (110) with respect to the accuracy (410).

6. The method according to any one of claims 1 to 5, wherein the calculation time is measured when evaluating the simulation of the behavior of the sensor (110) with respect to the calculation time (430).

7. The method according to any one of claims 1 to 6, wherein the second input value is generated by means of a sensor (110).

8. The method according to any one of claims 1 to 7, wherein a scenario for the first input value is specified depending on the second input value. 9 . A method for training a recognition algorithm, wherein the recognition algorithm is trained using simulated sensor data, wherein the simulated sensor data are evaluated using the method for evaluating a sensor model according to claim 1 .

10. A sensor system (100) having a sensor (110) and an evaluation unit (120), wherein the evaluation unit (120) is configured to execute a recognition algorithm, wherein the recognition algorithm is trained using the method for training a recognition algorithm according to claim 9.

Images