Method for evaluating software for vehicle control devices - Patents.com

Abstract

The present invention relates to a method for evaluating software (111) for a control device (102) of a vehicle (100), the control device (102) including a memory (110) in which a first version (111a) and a second version (111b) of the software (111) are stored, and a processor (112) for executing the first version (111a) and the second version (111b). The method includes the steps of receiving, in a control device (102), sensor data (108) generated by a sensor system (104) for sensing an environment of the vehicle (100), inputting the sensor data (108) into a first version (111a) and a second version (111b), generating first object data (206, 206') from the sensor data (108) by the first version (111a), the first object data (206, 206') including a position and / or orientation of a first object (116, 116') in the environment of the vehicle (100) as recognized by the first version (111a), and generating a second version (206, 206') from the sensor data (108) by the first version (111a), the first object data (206, 206') including a position and / or orientation of a first object (116, 116') in the environment of the vehicle (100) as recognized by the first version (111a). generating second object data (208, 208') from the sensor data (108) by the control device (102) via the first object data (206, 206'), the second object data (208, 208') comprising a position and / or orientation of the second object (118, 118') in the environment of the vehicle (100) as recognized by the second version (111b); evaluating the second version (111b) in terms of recognition quality by comparison between the first object data (206, 206') and the second object data (208, 208'), whereby an evaluation result (212) is generated; and transmitting the evaluation result (212) from the control device (102) to the central data processing device (214).

Description

[Technical field]

[0001] The present invention relates to a method for evaluating software for a control device of a vehicle. Furthermore, the invention relates to a control device, a computer program and a computer readable medium for carrying out said method. [Background technology]

[0002] A vehicle, such as, for example, a car or truck, may be equipped with a driver assistance system that enables partial or fully automatic control of the vehicle. To this end, the driver assistance system may recognize, for example using a suitable sensor system, the position, orientation, and / or object type of objects in the vehicle's environment and steer, brake, and / or accelerate the vehicle taking these objects into account.

[0003] Such driver assistance systems are generally subject to stringent security requirements. Updates to the driver assistance system, for example to improve or extend the driver assistance system, can be very costly as a release of the entire system must be performed every time an individual component is updated. Summary of the Invention

[0004] In view of the above background, the approach presented herein provides a method for evaluating software for a control device of a vehicle, a corresponding control device, a corresponding computer program and a corresponding computer-readable medium according to the independent claims. Advantageous developments and improvements of the approach presented herein become apparent from the description and are set forth in the dependent claims.

[0005] An embodiment of the present invention enables a not yet released version of object recognition software for a vehicle control device to be run on the control device in parallel with an already released version of the object recognition software, the not yet released version to be evaluated in terms of its recognition quality by comparing the recognition results of the two versions, and the corresponding evaluation results to be transmitted to a central data processing device for further analysis.

[0006] Thus, taking into account the evaluation results of a large number of corresponding series-produced vehicles, i.e. real data, a very fast and cost-effective validation of not yet released versions, e.g. new versions of object recognition or sensor data fusion modules, can be guaranteed.

[0007] A first aspect of the invention relates to a computer-implemented method for evaluating software for a control device of a vehicle, where the control device includes a memory in which a first version and a second version of the software are stored and a processor for executing the first version and the second version, the method comprising at least the steps of receiving in the control device sensor data generated by a sensor system for sensing an environment of the vehicle, inputting the sensor data into the first version and the second version, generating first object data from the sensor data by the first version, the first object data including a position and / or orientation of a first object in the environment of the vehicle recognized by the first version, generating second object data from the sensor data by the second version, the second object data including a position and / or orientation of a second object in the environment of the vehicle recognized by the second version, evaluating the second version with respect to recognition quality by comparison of the first object data and the second object data, where an evaluation result is generated, and transmitting the evaluation result from the control device to a central data processing device.

[0008] The method may be performed automatically, for example, by a processor of a control device, for example when instructions generated by a central data processing device for performing the method are received at the control device.

[0009] The vehicle may be, for example, a motor vehicle in the form of a car, a truck, a bus or a motorbike. In a broader sense, the vehicle may also be understood as an autonomously mobile robot. The sensor system may include at least one environmental sensor, such as a camera, a radar sensor, a lidar sensor, or an ultrasonic sensor. Furthermore, the sensor system may include a position sensor for determining the geographic coordinates of the vehicle using a global navigation satellite system, such as GPS or GLONASS. Additionally, the sensor system for sensing the driving state of the vehicle may include at least one driving dynamics sensor, such as an acceleration sensor, a wheel speed sensor, a steering angle sensor, a steering torque sensor, a brake pressure sensor, or a brake pedal travel sensor.

[0010] Processing of the sensor data in the control device allows recognition of objects in the vehicle's environment, such as other traffic participants, lane markings, traffic signs, traffic lights, buildings or vegetation, whereby the positions and / or orientations of the objects relative to the vehicle can be determined in multiple successive time steps and stored in the environment model in the form of an object list, whereby at each current time step the future positions and / or orientations of the objects can be estimated from their positions and / or orientations at one or more previous time steps.

[0011] The sensor data can be received at multiple successive time steps, and at each time step, the sensor data can be entered into both the first version and the second version.

[0012] The controller software can be configured to steer, accelerate, and / or brake the vehicle based on the sensor data. To this end, the vehicle can include corresponding actuator systems, for example in the form of steering actuators, braking actuators, engine controllers, electric drive motors, or a combination of at least two of the above mentioned examples. The controller software can include one or more components of a driver assistance system.

[0013] The central data processing device may be, for example, a server, a PC, a laptop, a tablet, or a smartphone. The control devices and the central data processing device can be connected to each other via a wireless data communication connection, for example WLAN, Bluetooth and / or mobile radio. However, a wired data communication connection between the control devices and the central data processing device is also possible.

[0014] The method may additionally include the steps of receiving a second version at the control device and storing the second version in a memory of the control device, where the second version may have been generated by and / or transmitted from the central data processing device to the control device. The second version may be received at and stored in the control device, for example in the form of data executable by a processor.

[0015] The first version may be an older, already released version of the software, and the second version may be a newer, not yet released version of the software.

[0016] For example, the second version may include updated versions of one or more software modules of the first version, such as, for example, a recognition or sensor data fusion module for recognizing objects in the sensor data, or an interpretation module for interpreting objects in terms of their relevance to the vehicle.

[0017] The first version may include a first recognition module for converting the sensor data into first object data and / or a first interpretation module for determining objects relevant to the vehicle based on the first object data and / or the sensor data. The first recognition module or the first interpretation module may be software modules of the already released software.

[0018] Similarly, the second version may include a second perception module for converting the sensor data into second object data and / or a second interpretation module for determining objects relevant to the vehicle based on the second object data and / or the sensor data. The second perception module or the second interpretation module may be software modules of the yet to be released software.

[0019] The first version and the second version may be executed in parallel, for example by a processor in a controller. Here, the second version may run in an isolated area within which the second version may run without security-related impacts to the first version of the vehicle or to hardware and software components of the vehicle outside of this area, such as the actuator systems, etc. Software running in such an isolated area may be referred to as shadow mode software.

[0020] In other words, the first version and the second version may execute in different operating environments, where the second version may execute in a more restricted operating environment than the operating environment of the first version.

[0021] For example, the first object data may include an object class of the first object in addition to the position and / or orientation of the first object. Alternatively or additionally, the second object data may include an object class of the second object in addition to the position and / or orientation of the second object. The object class may be, for example, an object type such as "oncoming vehicle", "preceding vehicle", "lane marking", "pedestrian", etc. An object class may be assigned to the first or second object by analysis of the sensor data using one or more classifiers. Multiple object classes may also be assigned to the same object.

[0022] To evaluate the second version with regard to recognition quality, the first and second object data can be compared with each other, for example by comparing the position and / or orientation of the first object with the position and / or orientation of the second object and / or by comparing the recognition time instant at which the first object was recognized with the recognition time instant at which the second object was recognized. However, other comparison methods are also possible.

[0023] For this purpose, a link between one first object and one second object in each case can be determined, where the positions and / or orientations and / or recognition times of the mutually linked objects can be compared with one another. The mutually linked objects can be object models representing the same real object in the vehicle's environment.

[0024] Evaluating the second version makes it possible to determine whether its recognition quality, i.e. the recognition accuracy and reliability with which the second version recognizes an object, is worse than the recognition quality of the first version or at least as good.

[0025] The evaluation results may include, for example, statistical estimates regarding the recognition accuracy and reliability of the second version, in absolute terms and / or relative to the first version. Additionally or alternatively, the evaluation results regarding the evaluation of the second version for recognition quality may include relevant data sequences from the first and / or second object data and / or sensor data. The data sequences may be analyzed at a central location, regardless of the current location of the vehicle, for example by transmitting the data sequences to a central data processing device via a wireless data communication connection.

[0026] Such a data sequence may include, for example, an object list including objects that are relevant with respect to the evaluation of the second version for recognition quality, and / or the underlying (raw) sensor data of these objects.

[0027] For example, when it is determined that an object recognized by a first version was not recognized by a second version, the second version may be evaluated as being worse than the first version in terms of recognition quality.

[0028] Conversely, when it is determined that an object recognized by the second version was not recognized by the first version, the second version can be evaluated as better than the first version in terms of recognition quality.

[0029] Here, it is advantageous to distinguish between vehicle-relevant and non-vehicle-relevant objects: for example, an object may be recognized as vehicle-relevant or non-vehicle-relevant depending on its distance and / or its relative speed with respect to the vehicle and / or depending on its object class.

[0030] The approach described here and below is based on the fact that software can be run in a production vehicle in a so-called shadow mode to assist in the development of driver assistance systems. Here, the software or individual software modules run passively in an isolated area without adversely affecting active components in the vehicle. In this area, for example, newly developed software versions can be run and evaluated in rapid iterations. For this purpose, for example, an evaluation framework can be used, which triggers the recording of data sequences based on a predefined trigger logic and ensures wireless data transmission to the cloud. Thus, newly developed software versions can be compared and evaluated very quickly using large amounts of data corresponding to real-world events in the field.

[0031] Now, in a method according to an embodiment of the first aspect of the invention, such a shadow mode can be utilized to determine whether an updated version of software for a control device of a vehicle achieves certain target parameters as well as or better than an already released version of this software that is active in the vehicle. For this purpose, the results of the updated version running in the shadow mode in the vehicle can be directly compared with the results of the already released version. This has the advantage that updates to software modules of, for example, a driver assistance system can be released at significantly less cost and therefore provided much more frequently compared to traditional release methods.

[0032] It should be noted that the first and second versions of the software do not necessarily need to run on the same control device to perform the methods described above and below: instead, the two versions may run on different control devices, possibly networked together, as described below.

[0033] It is possible for a first version to be executed by a first controller and a second version to be executed by a second controller, which may also execute the first (released) version of the software or another released version in addition to the second version.

[0034] The first controller may be, for example, a controller of a first vehicle. The second controller may be, for example, a controller of a second vehicle. The first controller and the second controller may be connected to each other for data communication, for example via a wireless data communication connection.

[0035] The first object data may be generated, for example, by a first control device from first sensor data, which may be generated by a first sensor system for sensing an environment of the first vehicle.

[0036] The second object data may, for example, be generated by a second control device from second sensor data, which may be generated by a second sensor system for sensing the environment of the second vehicle.

[0037] The first controller may receive second object data from the second controller. Thus, for example, an evaluation result can be generated by the first control device by comparison of the first object data with the second object data received from the second control device, whereby the evaluation result can be transmitted from the first control device to the central data processing device and additionally also to the second control device.

[0038] A second aspect of the invention relates to a control device including a processor configured to execute a method according to an embodiment of the first aspect of the invention. Features of the method according to an embodiment of the first aspect of the invention may also be features of the control device and vice versa.

[0039] The controller may include hardware and / or software modules. In addition to a processor, the controller may include a memory and a data communication interface for data communication with peripheral devices.

[0040] A third aspect of the invention relates to a computer program comprising instructions which, when the computer program is executed by a processor, cause the processor to perform a method according to an embodiment of the first aspect of the invention.

[0041] A fourth aspect of the present invention relates to a computer readable medium on which a computer program according to an embodiment of the third aspect of the present invention is stored. The computer readable medium may be a volatile or non-volatile data storage. For example, the computer readable medium may be a hard drive, a USB memory device, a RAM, a ROM, an EPROM, or a flash memory. The computer readable medium may also be a data communication network, such as the Internet or a data cloud (cloud), that allows the downloading of the program code.

[0042] Features of the method according to an embodiment of the first aspect of the invention may also be features of the computer program and / or the computer readable medium and vice versa. The ideas regarding the embodiments of the present invention can be considered to be based on the ideas and findings described below, among others.

[0043] According to an embodiment, for each first object, at least one first evaluation parameter is determined, which indicates how well the first object was recognized by the first version. Now, for each second object, at least one second evaluation parameter is determined, which indicates how well the second object was recognized by the second version. The second version is then evaluated by comparison of the first evaluation parameter with the second evaluation parameter. The first or second evaluation parameter can be, for example, a recognition time point or a statistical value, for example a confidence level for the recognized position and / or orientation. The second version can be evaluated by comparison of the first evaluation parameter with the second evaluation parameter of the same object. In other words, the first object and the second object that matches the first object can be evaluated by comparison of the first evaluation parameter assigned to the first object with the second evaluation parameter assigned to the second object. For example, for the comparison of the first version with the second version, a difference can be generated from the first evaluation parameter and the corresponding second evaluation parameter, and the second version can be evaluated based on the difference. This allows evaluation of the software based on individual recognized objects.

[0044] According to one embodiment, the first evaluation parameter is the recognition time when the first object is recognized by the first version. Additionally or alternatively, the second evaluation parameter may be the recognition time when the second object is recognized by the second version. For example, when the recognition time when the object is recognized by the second version is earlier than the recognition time when the same object is recognized by the first version, the second version can be evaluated as better in terms of recognition quality than the first version, and vice versa. Such a comparison of recognition times can be easily performed and provides a sufficiently accurate evaluation result.

[0045] According to an embodiment, the first evaluation parameter is a probability for the position and / or orientation of the first object. Additionally or alternatively, the second evaluation parameter may be a probability for the position and / or orientation of the second object. The probability may indicate for example the accuracy of the position and / or orientation, more precisely the accuracy of the position parameter for the probability distribution of the position and / or orientation. The first or second evaluation parameter may for example be a position parameter and / or a variance parameter of the probability distribution. It is also possible that the first or second evaluation parameter indicates a confidence interval. Thus, the accuracy and reliability of the method may be increased.

[0046] According to an embodiment, the first version selects from among the first objects a first object relevant to the vehicle. Here, the mutually matching objects are determined by a comparison of the relevant first and second objects. The evaluation parameters of the mutually matching objects are then compared with each other. As already mentioned further above, the relevant first object can be selected from among the first objects, for example depending on its distance and / or its relative speed with respect to the vehicle and / or depending on its object class. This can be done using a first interpretation module of the first version, which may be configured to determine the relevance of the first object depending on the situation and / or function, for example by dividing the first object into different relevance categories, in the simplest case for example into the relevance categories "relevant" and "irrelevant". This makes it possible to determine whether the second version even recognizes the object recognized as relevant by the first version (which has been verified or released). If this is determined, this can be evaluated as a strong indication that the recognition quality of the second version is at least not worse than the recognition quality of the first version.

[0047] According to an embodiment, the second version selects from the second objects second objects that are relevant to the vehicle and do not match any of the relevant first objects. Here, an individual evaluation is generated for each relevant second object, the individual evaluation indicating whether the recognition of the relevant second object by the second version corresponds to an improvement or a deterioration of the recognition quality of the second version with respect to the first version. The second version is then further evaluated based on the individual evaluation. For this purpose, it can first be determined whether the second object data includes a second object that does not match any of the relevant first objects, i.e. that was not recognized in the first version or at least was not recognized as relevant. It can also be further determined whether the second object that does not match any of the relevant first objects is relevant to the vehicle. As already mentioned further above, this can be done using a second interpretation module of the second version which, like the first interpretation module, may be configured to determine the relevance of the second object depending on the situation and / or function, e.g. by dividing the second object into various relevance categories, in the simplest case e.g. into the relevance categories "relevant" and "not relevant".

[0048] The individual evaluations can be transmitted to a central data processing device, for example as part of an evaluation result. Additionally or alternatively, the evaluation result can include the object data and / or sensor data on which the respective individual evaluation is based. The object data and / or sensor data can be transmitted as part of the evaluation result only when the individual evaluation on which they are based indicates a deterioration in the recognition quality of the second version relative to the first version.

[0049] This embodiment allows the second version to be rated depending on whether it recognizes relevant objects not already recognized by the first version, and the rating of the second version can for example be recalculated with each individual evaluation.

[0050] According to one embodiment, based on the sensor data and / or driving dynamics data generated by at least one driving dynamics sensor of the vehicle, changes in the driving state of the vehicle are determined, which changes are time-correlated with the recognition time at which the relevant second object was recognized by the second version. Here, each individual evaluation is generated by analysis of the changes in the driving state that are time-correlated with the respective relevant second object. In order to determine whether the recognition of a relevant second object that was not recognized (as relevant) by the first version actually corresponds to an improvement in the recognition quality of the second version relative to the first version, the sensor data and / or driving dynamics data can for example be analyzed to determine and interpret the reaction of the driver of the vehicle at the recognition time of the object in question. For example, if the driver's reaction, or at least a relevant reaction, cannot be determined, this can be evaluated as a strong indication that the recognition quality was not significantly improved by the recognition of the object in question, and vice versa.

[0051] According to one embodiment, the second object data is generated in a number of consecutive time steps. Here, the second object is checked for validity by comparing the second object data from different time steps. The second version is then further evaluated according to the validity of the second object. For example, the positions and / or orientations of the same object from different time steps can be compared with each other to determine inconsistencies, i.e., invalid changes in the object's position and / or orientation. This can determine whether the second version provides valid object data that is consistent in time. The object data for each invalid object, such as their positions and / or orientations over a number of consecutive time steps, can be transmitted to a central data processing device as part of the evaluation result.

[0052] According to one embodiment, the evaluation result comprises data sequences from the sensor data, the first object data and / or the second object data. The evaluation of the second version can be based on the data sequences. In other words, the data sequences can indicate an improvement or a deterioration in the recognition quality of the second version with respect to the first version. The transmission of these data sequences enables an intended external analysis of the second version with respect to its recognition quality.

[0053] According to one embodiment, the data sequence is transmitted only if the second version is evaluated as being worse than the first version in terms of recognition quality, thereby making it possible to improve the efficiency of data communication between the control device and the central data processing device.

[0054] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings, however, neither the drawings nor the description should be construed as limiting the present invention. [Brief description of the drawings]

[0055] [Figure 1] 1 illustrates a vehicle equipped with a control device according to an exemplary embodiment of the present invention. [Diagram 2] FIG. 2 illustrates various modules of software running on the controller from FIG. 1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0056] The figures are only schematic and are not drawn to scale. In the different figures, the same reference signs indicate the same features or features with the same function. FIG. 1 shows a vehicle 100 equipped with a control device 102, a sensor system 104 for sensing the environment of the vehicle 100, and an actuator system 106.

[0057] The sensor system 104 may include, for example, a camera, a radar sensor, a lidar sensor, and / or an ultrasonic sensor. Additionally, the sensor system 104 may include at least one driving dynamics sensor, for example an acceleration sensor or a yaw rate sensor. The sensor system 104 generates sensor data 108 at multiple successive time steps, which are received by the control device 102 and analyzed therein within the framework of object recognition. The control device 102 includes a memory 110 in which corresponding software 111 is stored and a processor 112 for executing the software 111.

[0058] Based on the sensor data 108, i.e. based on the results of the object recognition performed thereby, the control device 102 is capable of generating control signals 114 for automatic control of the actuator system 106. The actuator system 106 may include, for example, one or more steering and / or braking actuators for steering or braking the vehicle 100.

[0059] In the control device 102, two different versions 111a, 111b of the software 111 are executed at the same time, and the versions 111a, 111b are respectively input with sensor data 108. Here, an analysis of the sensor data 108 by the first version 111a of the software 111 recognizes a first object 116 in the environment of the vehicle 100, and an analysis of the sensor data 108 by the second version 111b of the software 111 recognizes a second object 118 in the environment of the vehicle 100. The first object 116 and the second object 118, here for example a preceding vehicle, may be the same object or different objects. By comparing the first object 116 and the second object 118, the control device 102 can evaluate the recognition quality of the second version 111b with respect to the first version 111a. This is described in more detail below based on FIG. 2.

[0060] 2 shows a block diagram of the software 111 running on the control device 102. In this example, the first version 111a is already verified or released, i.e., the active version of the software 111, and the second version 111b is a not yet verified or not yet released version of the software 111, which, unlike the first version 111a, may run within the restricted operating environment 200 of the control device 102, i.e., may run together with the first version 111a passively in a test area shielded from components outside the restricted operating environment 200. The second version 111b may be an update of the first version 111a, e.g., an improved and / or extended version relative to the first version 111a of the software 111. The second version 111b may also be referred to as shadow software.

[0061] Alternatively, the second version 111b may be assigned to a separate controller as well. In this example, the sensor data 108 enters both the first recognition module 202 of the first version 111a and the second recognition module 204 of the second version 111b. Here, the first recognition module 202 generates first object data 206 from the sensor data 108, the first object data 206 including the position and / or orientation of the first object 116 relative to the vehicle 100. Here, the second recognition module 204 generates second object data 208 from the sensor data 108, the second object data 208 including the position and / or orientation of the second object 118 relative to the vehicle 100.

[0062] The recognized objects 116, 118 together with their respective positions and / or orientations, for example in the form of an object list, can be stored in an environment model of the environment of the vehicle 100, where it can be continuously updated based on the sensor data 108. In this context, the first object 116 and the second object 118 can be understood as object models of objects actually present in the environment of the vehicle 100, stored in the environment model.

[0063] In addition to the respective position and / or orientation, the first object data 206 and / or second object data 208 may specify one or more object types, such as “vehicle,” “pedestrian,” or “lane marking,” for each recognized object 116 or 118.

[0064] The first object data 206 and the second object data 208 are input to an evaluation module 210 of the software 111, which in this example runs in a restricted operating environment 200, as well as the second version 111b, for security reasons. The evaluation module 210 evaluates the recognition quality of the second version 111b relative to the first version 111a by appropriately comparing the first object data 206 with the second object data 208. The evaluation module 210 now generates a corresponding evaluation result 212 and transmits the evaluation result 212 to a central data processing device 214 outside the vehicle 100, for example via a WLAN connection, a Bluetooth connection and / or a mobile radio connection, for further analysis.

[0065] Within this evaluation framework, first objects 116 can be compared with corresponding second objects 118 based on one or more suitable evaluation parameters, for example based on the respective recognition time points or based on one or more estimates regarding the accuracy and / or reliability of the respective positions and / or orientations.

[0066] The first object data 206 can be interpreted by the first interpretation module 216 of the first version 111a, i.e. analyzed with respect to their relevance with respect to the vehicle 100. The first interpretation module 216 can now select one or more relevant first objects 116′ from among the first objects 116 depending on the current situation of the vehicle 100 (e.g. a leading vehicle on the left side of the vehicle 100 in FIG. 1 ) and transmit the correspondingly filtered first object data 206′ to the evaluation module 210.

[0067] The evaluation module 210 can then link the relevant first objects 116' to the corresponding second objects 118, for example based on their respective positions and / or orientations and / or their respective recognition times. The evaluation parameters of the linked objects can then be compared to each other to evaluate the second version 111b. The linked objects may be matching objects insofar as they are object models of the same object actually present in the environment of the vehicle 100.

[0068] Furthermore, the second object data 208 can be interpreted by the second interpretation module 218 of the second version 111b, i.e. analyzed with respect to their relevance with respect to the vehicle 100. Here, the second interpretation module 218 can select one or more relevant second objects 118′ from among the second objects 118 depending on the current situation of the vehicle 100 (e.g. a leading vehicle on the right side of the vehicle 100 in FIG. 1 ) and transmit the correspondingly filtered second object data 208′ to the evaluation module 210.

[0069] In this case, the evaluation module 210 can determine whether there are any objects among the relevant first objects 206' and the relevant second objects 208' that uniquely match each other. If one of the relevant second objects 208' does not match one of the relevant first objects 206', the evaluation module 210 can generate an individual evaluation for this second object, indicating whether the recognition of this second object represents an improvement or a deterioration in the recognition quality of the second version 111b relative to the first version 111a. An evaluation result 212 can then be generated based on the individual evaluation.

[0070] Generation of the personal assessment may be based on sensor data 108, where sensor data 108 may include driving dynamics data generated by one or more driving dynamics sensors of vehicle 100 in addition to environmental data generated by one or more environmental sensors of vehicle 100.

[0071] Upon analysis of the sensor data 108 by the evaluation module 210, a change in the driving dynamics state of the vehicle 100 can be determined that correlates in time with the recognition of the relevant second object 118', e.g. caused by a corresponding reaction of the driver of the vehicle 100. Based on this change, it can finally be determined whether the recognition of said object corresponds to an improvement or a deterioration in the recognition quality of the second version 111b.

[0072] Alternatively, the second interpretation module 218 may be configured to first determine second objects 118 that do not uniquely match any of the relevant first objects 116' and subsequently select a relevant second object 118' therefrom.

[0073] Additionally, the assessment module 210 may be configured to check the validity of the second object 118. To this end, the assessment module 210 may analyze the second object 118 based on the second object data 208 of multiple consecutive time steps, whereby the assessment result 212 may further be determined taking into account the validity of the second object 118. An example of an invalid or inconsistent second object 118 is suggested in FIG. 1 by a dashed box.

[0074] The evaluation of the second version 111b in the sense of validating the recognition task to be solved using the second recognition module 204 and / or the second interpretation module 218 may include, for example, the following steps.

[0075] Firstly, it is checked whether the relevant first object 116' recognized by the first version 111a was recognized similarly or in a better manner by the second version 111b executed in shadow mode. Here, the relevance of the first object 116 is not determined by the first recognition module 202 itself, but by the first interpretation module 216, i.e. by one or more subsequent interpretation software elements, in a kind of situation analysis. In order to determine which of the second objects 118 corresponds to the relevant first object 116', a linking is performed in the evaluation module 210. The recognition qualities of the two versions 111a, 111b are then compared with each other based on predefined metrics, which may include, for example, confidence regarding the respective recognition times or the respective positions and / or orientations. If a deterioration in the recognition quality of the second version 111b is determined, a corresponding data sequence can be directly transmitted to the central data processing device 214. Here, the data sequence can be generated from the corresponding sensor data 108 and / or the corresponding object data 116 or 118. Conversely, an improvement in recognition quality may be confirmed by the failure of such data sequences to arrive at the central data processor 214. Additionally or alternatively, the controller 102 may detect an improvement or deterioration in recognition quality by periodically transmitting compiled statistics to the central data processor 214.

[0076] Secondly, for each second object 118 that is recognized by the second version 111b but cannot be linked to one of the relevant first objects 116', its relevance with respect to the vehicle 100 is determined and an evaluation is performed indicating whether the recognition of this second object has worsened or improved the recognition quality. If the second interpretation module 218, in a kind of situation analysis, determines a second relevant object 118' that cannot be linked to any of the relevant first objects 116', firstly, using a predefined logic, it is determined whether the recognition of this object represents an improvement or a deterioration of the recognition quality, for example based on the driver's reaction to this object. Depending on the logic, a statistical process can be detected. Additionally or alternatively, a direct transmission of the corresponding data sequence can be triggered for external analysis in the central data processing device 214.

[0077] Thirdly, it is checked whether the second objects 118 and / or the relevant second objects 118' are consistent and plausible over time. For this purpose, the evaluation module 210 detects inconsistencies, such as the sudden appearance or disappearance of the second objects 118 in the immediate environment of the vehicle 100, based on the time profiles from the sensor data 108 and / or the second object data 208 or 208'. Information about these objects can be transmitted from the control device 102 to the data processing device 214 directly in the form of corresponding data sequences or collected in the form of statistics.

[0078] Validation of the classification task can be performed similarly. Finally, it should be noted that terms such as "comprise" and "include" do not exclude other elements or steps, and terms such as "eine" or "ein" do not exclude a plurality. Reference signs in the claims are not to be interpreted as limiting.

Claims

1. A method for evaluating software (111) for a control device (102) of a vehicle (100), comprising the steps of: the control device (102) includes a memory (110) in which a first version (111a) and a second version (111b) of the software (111) are stored, and a processor (112) for executing the first version (111a) and the second version (111b); receiving, in the control device (102), sensor data (108) generated by a sensor system (104) for sensing an environment of the vehicle (100); inputting said sensor data (108) into said first version (111a) and said second version (111b); generating first object data (206, 206′) from the sensor data (108) by the first version (111a), the first object data (206, 206′) comprising a position and / or an orientation of a first object (116, 116′) in the environment of the vehicle (100) as recognized by the first version (111a); generating second object data (208, 208') from the sensor data (108) by the second version (111b), the second object data (208, 208') comprising a position and / or an orientation of a second object (118, 118') in the environment of the vehicle (100) as recognized by the second version (111b); - evaluating the second version (111b) with respect to recognition quality by comparison of the first object data (206, 206') with the second object data (208, 208'), generating an evaluation result (212); transmitting said evaluation results (212) from said control device (102) to a central data processing device (214); Including, the second object data (208, 208') is generated at a plurality of successive time steps; the second object (118, 118') is checked for validity by comparing the second object data (208, 208') from different time steps; The second version (111b) is further evaluated depending on the validity of the second object (118, 118').

2. for each first object (116, 116') at least one first evaluation parameter is determined, which is indicative of how well said first object (116, 116') has been recognized by said first version (111a); for each second object (118, 118') at least one second evaluation parameter is determined, which is indicative of how well said second object (118, 118') has been recognized by said second version (111b); The method of claim 1 , wherein the second version (111b) is evaluated by a comparison of the first evaluation parameter and the second evaluation parameter.

3. A method for evaluating software (111) for a control device (102) of a vehicle (100), comprising: the control device (102) includes a memory (110) in which a first version (111a) and a second version (111b) of the software (111) are stored, and a processor (112) for executing the first version (111a) and the second version (111b); receiving, in the control device (102), sensor data (108) generated by a sensor system (104) for sensing an environment of the vehicle (100); inputting said sensor data (108) into said first version (111a) and said second version (111b); generating first object data (206, 206′) from the sensor data (108) by the first version (111a), the first object data (206, 206′) comprising a position and / or an orientation of a first object (116, 116′) in the environment of the vehicle (100) as recognized by the first version (111a); generating second object data (208, 208') from the sensor data (108) by the second version (111b), the second object data (208, 208') comprising a position and / or an orientation of a second object (118, 118') in the environment of the vehicle (100) as recognized by the second version (111b); - evaluating the second version (111b) with respect to recognition quality by comparison of the first object data (206, 206') with the second object data (208, 208'), generating an evaluation result (212); transmitting said evaluation results (212) from said control device (102) to a central data processing device (214); Including, for each first object (116, 116') at least one first evaluation parameter is determined, which is indicative of how well said first object (116, 116') has been recognized by said first version (111a); for each second object (118, 118') at least one second evaluation parameter is determined, which is indicative of how well said second object (118, 118') has been recognized by said second version (111b); said second version (111b) being evaluated by a comparison of said first evaluation parameters with said second evaluation parameters; said first version (111a) selects from among said first objects (116) a first object (116') that is relevant to said vehicle (100); matching objects (116', 118, 118') are determined by comparing the relevant first object (116') with the relevant second object (118, 118'); The evaluation parameters of the matching objects (116', 118, 118') are compared with each other.

4. said first evaluation parameter being the recognition time at which said first object (116, 116') was recognized by said first version (111a); and / or The method according to claim 2 or 3, wherein the second evaluation parameter is a recognition time instant at which the second object (118, 118') was recognized by the second version (111b).

5. the first evaluation parameter is a probability for the position and / or orientation of the first object (116, 116'); and / or The method according to any one of claims 2 to 4, wherein the second evaluation parameter is a probability for the position and / or orientation of the second object (118, 118').

6. said second version (111b) selects from among said second objects (118) a second object (118') associated with said vehicle (100) that does not match any of said associated first objects (116'); an individual assessment is generated for each relevant second object (118'), said individual assessment indicating whether the recognition of said relevant second object (118') by said second version (111b) corresponds to an improvement or a deterioration of the recognition quality of said second version (111b) relative to said first version (111a); the second version (111b) is further evaluated based on the individual evaluation. The method according to claim 3.

7. Based on the sensor data (108) and / or driving dynamics data generated by at least one driving dynamics sensor of the vehicle (100), changes in driving conditions of the vehicle (100) are determined, said changes being correlated in time with a recognition time at which the relevant second object (118′) was recognized by the second version (111b); each individual assessment is generated by analyzing the changes in the driving conditions correlated in time with each relevant second object (118'); The method according to claim 6.

8. The method of any one of claims 1 to 7, wherein the evaluation result (212) comprises data sequences from the sensor data (108), the first object data (206, 206'), and / or the second object data (208, 208').

9. said data sequence being transmitted only if said second version (111b) is assessed as being worse than said first version (111a) in terms of said recognition quality. The method according to claim 8.

10. A control device (102) including a processor (112) configured to execute the method of any one of claims 1 to 9.

11. A computer program executed by the processor (112), comprising instructions causing the processor (112) to perform the method of any one of claims 1 to 9.

12. A computer readable medium having stored thereon the computer program of claim 11.

Images