Computer-implemented method and system for validating a behaviour planner for an at least partially automated vehicle

Abstract

The invention proposes a computer-implemented method for validating a behaviour planner for an automated vehicle. The validation method according to the invention uses a database (31) containing predefined partial situations and at least one evaluation model for each partial situation, wherein the partial situations of the database (31) are combinable to form different complete situations, and boundary conditions for permissible behaviour options of the vehicle in such a complete situation are able to be determined on the basis of the evaluation models of the contributing partial situations. According to the invention, test scenarios (32) are created for validating the behaviour planner (30), specifically in the form of a sequence of test situations, wherein each test situation is formed in the form of a composition of partial situations (20) of the database (31). The results of the behaviour planner (30) for the test situations of the individual test scenarios (32) are then analysed with the aid of the evaluation models (27) of the contributing partial situations (20).

Description

[0001] Description

[0002] title

[0003] Computer-implemented method and system for validating a behavior planner for an at least partially automated vehicle

[0004] State of the art

[0005] The invention relates to a computer-implemented method and a system for validating a behavior planner for an at least partially automated vehicle, which is also referred to below as ego vehicle.

[0006] In the publication, M. Butz et al., "SOCA: Domain Analysis for Highly Automated Driving Systems," 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), 2020, pp. 1 -6, doi:

[0007] 10.1109 / ITSC45102.2020.9294438., the so-called SOCA method is presented. Using the SOCA method, traffic situations are analyzed with the aim of determining boundary conditions for or requirements regarding the behavior of an automated ego-vehicle in the respective traffic situation. To do this, an abstract description of the traffic situation to be analyzed is first created. This description uses so-called zone graphs. A zone graph abstracts the traffic situation to be analyzed by representing the real road situation using a corresponding abstract traffic infrastructure element (static road geometry) with different zones that are relevant to the driving intention of the ego-vehicle but are initially not specified in terms of their size or position. The different zones can represent different map areas, possible traffic flows, objects, etc.Based on this abstract description of the traffic situation, the possible developments or behavior of the road users involved are then determined and morphologically analyzed to determine boundary conditions for the behavior of the ego vehicle in the analyzed traffic situation. It is noteworthy that the results or boundary conditions obtained in this way initially apply to all traffic situations with the same zone graph. A specification is only achieved by conditioning the results with the situation-specific parameters of the analyzed traffic situation.

[0008] DE 10 2022 214 276 describes a behavior planner that determines boundary conditions for permissible behavior options of the ego vehicle in a given situation. This behavior planner works with a database of predefined partial situations. The database also provides at least one evaluation model for each of these partial situations. First, the given situation is broken down into partial situations in the database. The boundary conditions for permissible behavior options of the ego vehicle in the given situation are then determined using the associated evaluation models, namely as a combination of boundary conditions of the individual partial situations.

[0009] During the validation of a behavior planner, the ego vehicle is typically exposed to a large number of different situations and scenarios, i.e., a sequence of situations, in order to test the ego vehicle's behavior proposed by the behavior planner and evaluate it against predefined criteria. These scenarios are referred to below as test scenarios.

[0010] An example of a test scenario consisting of a sequence of three situations is shown in Figs. 1a to 1c. The test scenario relates to a four-way intersection. In the first (traffic) situation - Fig. 1a - three vehicles 1, 2, 3 approach the intersection area 10 from different directions: the ego vehicle 1 from the south, vehicle 2 from the north, and vehicle 3 from the east. According to the signs 13, 14, vehicle 3 is moving on a priority road 11, while the ego vehicle 1 and vehicle 2 must observe the right of way on the intersecting road 12. Therefore, stop lines 121, 122 are marked on the carriageways of road 12, each in the direction of travel before the intersection area 10.

[0011] In the second (traffic) situation - Fig. 1 b - vehicle 3 has entered the intersection 10, while the ego vehicle 1 and vehicle 2 have stopped at the stop line 121 and 122, respectively, to yield to vehicle 3. In the third (traffic) situation - Fig. 1 c - vehicle 3 has passed the intersection 10 and continues its journey westward on the priority road 12. The ego vehicle 1 is still at the stop line 121, while vehicle 2 has already entered the intersection 10 to turn right westward onto the priority road 12.

[0012] Fig. 1a to 1c illustrate that each (traffic) situation of a test scenario is characterized by a number of road users and their movement states. In addition to the ego vehicle 1, two other vehicles 2 and 3 are located at different positions and at different speeds in the intersection environment.

[0013] Disclosure of the invention

[0014] The present invention proposes measures for automatically generating test scenarios that support systematic testing of any type of behavior planner. These measures are particularly suitable for validating behavior planners for a given operational design domain (ODD).

[0015] This is achieved according to the invention by providing a database with predefined partial situations and at least one evaluation model for each partial situation. The partial situations of the database can be combined to form different overall situations, and boundary conditions for permissible behavior options of the vehicle in such an overall situation can be determined based on the evaluation models of the contributing partial situations. According to the invention, test scenarios are generated in the form of a sequence of test situations, with each test situation being formed in the form of a composition of partial situations of the database. The results of the behavior planner for the test situations of the individual test scenarios are analyzed using the evaluation models of the contributing partial situations.

[0016] The inventive approach uses an abstract scenario concept to systematically generate concrete test scenarios. This scenario concept stipulates, on the one hand, that each scenario consists of a sequence of situations, and, on the other hand, that each of these situations can be represented as a composition of predefined and pre-analyzed partial situations. According to the invention, the concept of partial situations is therefore used to construct test situations and test scenarios for the purpose of validation. Accordingly, the composition of the database is important. The partial situations of the database should be selected and configured such that, taken as a whole, all possible scenarios of an ODD of the ego vehicle can be generated. Only then can a meaningful and meaningful coverage of the ODD with test scenarios be achieved.Since the test situations of these test scenarios are generated according to the invention as a composition of partial situations of the database, boundary conditions for permissible behavior options of the ego vehicle can be determined for each test situation with the help of the corresponding evaluation models.

[0017] According to the invention, these boundary conditions form the basis for validation. When analyzing the results determined by the behavior planner to be validated for the individual test situations of a test scenario, it is checked whether the boundary conditions for permissible behavior options of the ego vehicle are met in the respective test situation. The application of the method according to the invention is therefore independent of the implementation of the behavior planner. It can be used to validate behavior planners of a wide variety of implementations, with the goal of validation here being to check whether the behavior planner complies with the boundary conditions for permissible behavior options in the generated test scenarios.

[0018] The validation method according to the invention enables the systematic formalization and exploration of a given ODD, e.g., as a release argument for a behavior planner. Furthermore, it can be used for the automated detection of critical situations within a given behavioral framework of the ego vehicle and for the derivation of corresponding test scenarios.

[0019] As already mentioned, the database of predefined partial situations should be compiled in such a way that many different traffic situations can be completely broken down into partial situations in the database, in particular, as many operating conditions and deployment situations of the ego vehicle as possible, as defined by the ODD. This means that the database must contain a corresponding variety of partial situations. The partial situations should, on the one hand, be generic enough to occur in as many different traffic situations as possible, but, on the other hand, also specific enough that the associated evaluation model has a certain degree of meaningfulness.Furthermore, the individual partial situations should be “tailored” in such a way that the associated evaluation model takes into account as many influencing factors on the behavior of the ego vehicle as possible and specifies corresponding boundary conditions for the permissible behavior of the ego vehicle in this partial situation.

[0020] In principle, there are different options for defining partial situations and implementing and presenting the corresponding evaluation models.

[0021] In a preferred embodiment of the invention, each partial situation is defined as a situation class that is at least partially determined by an ego vehicle with a behavior model, at least one traffic infrastructure element, and a general situation context. Optionally, the definition of a partial situation can also include at least one additional road user with a behavior model. Each partial situation and the associated evaluation model can be parameterized for a specific situation and / or instantiated by loading it with situation-specific information.

[0022] In this embodiment of the invention, generating a test situation includes not only selecting suitable partial situations from the database, but also defining parameterizations and / or instantiations of the selected partial situations or evaluation models. The partial situations defined in this way can be combined very flexibly to construct complex situations. For example, multiple instances of the same partial situation with different parameterizations and / or data assignments can be used to construct a complex situation. An example of such a complex situation is shown in Figs. 1a to 1c. It can be easily constructed from two instances of a partial situation, each of which is determined by an ego vehicle with a behavior model, a four-way intersection where right-of-way is controlled by signage, and another vehicle.In a first instance of this partial situation, the other vehicle is the oncoming vehicle 2. A second instance of this partial situation treats vehicle 3 on the priority road 11 as another vehicle.

[0023] A very effective method for analyzing traffic situations with the goal of determining boundary conditions for the behavior of an automated ego-vehicle in the respective traffic situation is the SOCA method described above. The SOCA method is also suitable for defining partial situations and generating evaluation models for partial situations, as used in the invention. In this case, the evaluation models are based on a decomposition of the respective partial situation into zone graphs and a morphological behavior analysis of the road users involved. The evaluation models are generated here in the form of Zwicky boxes. A Zwicky box is a fully formalized description of the boundary conditions or restrictions for the permissible behavior of the ego-vehicle in the respective partial situation, for all reasonable instantiations of this partial situation.

[0024] The Zwicky boxes generated using the SOCA method—like the associated partial situations—can be combined very easily and flexibly to determine boundary conditions for the permissible behavior of the ego vehicle in complex situations constructed in this way. The combination of evaluation models of multiple partial situations of a test situation must take into account the fact that some partial situations or instances of partial situations can interact and may even be mutually exclusive. In a preferred embodiment of the invention, abstract test scenario classes are first created to generate test scenarios, based on a predefined scenario specification. A test scenario class consists of a sequence of test situation classes, the number of which is specified by the scenario specification.Furthermore, the scenario specification determines which partial situations from the database are composed into the individual test situation classes. In addition to the selection, the scenario specification also specifies the instantiation of the partial situations and, if necessary, at least partial parameterization of the partial situations or test situation classes.

[0025] In the simplest case, such a scenario specification is a declarative specification of an event chain. This merely specifies states for the individual situations of the scenario, while the necessary operations to reach these states are not specified. For example, the situation depicted in Fig. 1 b can be declaratively specified as a situation in which the ego vehicle is waiting at a four-way intersection while another vehicle 2 is also waiting on the opposite street of the intersection and a third vehicle 3 is currently crossing the intersection on the priority road 11. Such declarative specifications are supported, for example, by the new OpenScenario 2.0 standard and are expected to be supported by tools in the near future.

[0026] Different concrete test scenarios can then be easily generated from a test scenario class by fully parameterizing and / or data-encoding the test situation classes of this test scenario class. In this case, predefined boundary conditions for the test scenarios can advantageously also be taken into account, in particular general boundary conditions for the development of a traffic scene and / or ODD boundary conditions that define the ODD of the behavior planner. General boundary conditions describe, for example, physical restrictions for the development of a traffic scene. For example, it can be taken into account that a road user always requires a minimum amount of time to move from a first position in the test scenario to a second position in the traffic scene that is spaced apart from the first position.Using ODD boundary conditions, the test scenarios can be limited to a specific ODD, for example to certain infrastructure elements, such as a highway, or certain geographical areas, such as a closed logistics center.

[0027] In a further advantageous embodiment of the method according to the invention, a number and / or selection of test scenarios that satisfy a predetermined coverage criterion is generated for each test scenario class. Thus, a set of test scenarios is generated for each test scenario class that covers the "space" of the respective test scenario class according to the coverage criterion.

[0028] The method according to the invention offers the possibility of automatically and systematically generating test scenarios for validating the behavior planner. In this context, it proves particularly advantageous if the partial situations of the database as well as the corresponding evaluation models, the scenario specifications, the general constraints, and also the ODD constraints are provided in a formally explorable, declarative program representation. The test scenarios can then be automatically generated from this declarative program representation using a solver tool.

[0029] It has been recognized that methods from the field of declarative problem-solving, which have been developed in the context of knowledge representation, logic programming, satisfiability solving, and database theory, can also be advantageously used within the framework of the method according to the invention. In declarative problem-solving methods, complex tasks are described in the form of a declarative program representation, which enables the use of highly efficient software tools for finding solutions, so-called solver tools, for the described tasks. Furthermore, declarative program representations of subtasks can be easily combined, so that the resulting combined program representation can then be used as the basis for solving the overall problem.For the inventive use of these methods, it proves advantageous that the evaluation models of the partial situation are usually already available in declarative form. The evaluation models can then either be used directly or very easily converted into a declarative program representation. The invention also exploits the fact that the solutions generated in this way are formally explorable. This allows the solutions—here, the boundary conditions for permissible vehicle behavior in the individual test situations of the test scenarios—to be analyzed automatically, which is essential for comprehensive validation of the behavior planner based on the test scenarios.

[0030] A particularly suitable declarative program representation for the invention is answer set programs, as described, for example, in the publication by G. Brewka, T. Eiter, M. Truszczynski, "Answer Set Programming at a Glance" in CACM, Dec 2011, Vol. 54, No. 12, pp. 93-103. Answer set programming (ASP) is a declarative problem-solving paradigm for which a number of highly efficient ASP solvers have already been developed and are therefore available. These can be used to model complex tasks, such as test scenarios, and to find solutions for these tasks.

[0031] As already mentioned, when using a declarative program representation for the partial situations of the database, the evaluation models, the scenario specification, the general boundary conditions, and the ODD boundary conditions, solver tools can be used to automatically and specifically generate test scenarios. For this purpose, a series of restrictions for the abstract scenarios are specified in the scenario specification, such as a chronologically ordered sequence of potentially compound situations. An example of this is shown in Figs. 1a to 1c. With the help of a solver tool, all possible abstract scenarios can then be found that satisfy these restrictions while observing general boundary conditions. In the case of Figs. 1a to c, for example, it can be excluded that the vehicles move backward through their respective zones.This example illustrates that such scenario specifications can usually be formulated simply. However, it is generally difficult and error-prone to define a set of abstract scenarios in such a way that it satisfies such a scenario specification and ensures reasonable test coverage with a manageable number of test scenarios. During validation, particularly challenging situations or scenarios for the autonomous vehicle that arise rather unexpectedly should be investigated.

[0032] An advantageous embodiment of the method according to the invention provides for the validation of the behavior planner to be based on an optimized set of test scenarios. Optimized in this context means that this set of test scenarios satisfies a predefined validation criterion, such as a coverage criterion for a complete ODD. The optimized set of test scenarios is generated iteratively by varying the scenario specification and / or the selection of the predefined general constraints and / or the ODD constraints and / or the coverage criterion in each iteration step until the resulting set of test scenarios satisfies the validation criterion.If only the scenario specification and / or the selection of the predefined general constraints and / or the ODD constraints are varied in the individual iteration steps, then the test scenario class, i.e., the model itself, is varied. If only the coverage criterion is varied, i.e., the number and selection of test scenarios per test scenario class, then the model is not changed; instead, additional test scenarios are simply generated to iteratively explore the test scenario space of a test scenario class.

[0033] In any case, an iterative approach requires a query / refinement cycle with the system, alternating user input and automated generation of scenario sets until the user is satisfied with the query result.

[0034] This approach relies on manual specification to create an optimized set of test scenarios, but only in a complementary manner and to support the automatic generation of test scenarios.

[0035] As already explained at the beginning, the method according to the invention can in principle be used to validate any behavior planner for at least partially autonomous vehicles, as long as a suitable database with predefined partial situations and corresponding evaluation models is available. One possible application is the validation of behavior planners that have such a database and use it to break down a given situation into partial situations of the database and, based on the associated evaluation models, to determine boundary conditions for permissible behavior options of the vehicle in the given situation. In this case, the behavior planner's database can also be used to generate the test scenarios.

[0036] In addition to the validation methods described above, a computer-implemented system for validating a behavior planner for an at least partially automated vehicle is also claimed, which is particularly designed to execute the claimed validation method. Accordingly, the system according to the invention comprises a database with predefined partial situations and at least one evaluation model for each partial situation. The partial situations of the database can be combined to form different overall situations, and boundary conditions for permissible behavior options of the vehicle in such an overall situation can be determined based on the evaluation models of the contributing partial situations.Furthermore, at least one interface is provided for specifying scenario specifications, at least one composition tool for generating test scenario classes with test situation classes based on scenario specifications, and at least one solver tool for generating individual test scenarios of a test scenario class, whereby a test scenario is formed in the form of a sequence of test situations. Finally, the system according to the invention also comprises at least one analysis tool for analyzing the results of the behavior planner for the test situations of the individual test scenarios using the evaluation models of the contributing partial situations.

[0037] In an advantageous embodiment of the system according to the invention, a database is also provided in which predefined boundary conditions for the test scenarios are stored, in particular general boundary conditions for the development of a traffic scene and / or ODD boundary conditions that define the ODD of the behavior planner, so that these boundary conditions can be selectively retrieved and taken into account when generating the test scenarios. In the simplest case, these predefined boundary conditions for the test scenarios are stored in the same database as the predefined partial situations and the associated evaluation models. However, a second database can also be provided specifically for the predefined boundary conditions for the test scenarios.

[0038] Advantageously, a storage medium is also provided for storing the test scenario classes with associated evaluation models and / or for storing the generated test scenarios. This allows a database of test scenarios to be created that can be used repeatedly for the validation of behavior planners.

[0039] Drawings

[0040] Embodiments and advantageous developments of the invention are explained in more detail below in conjunction with the figures.

[0041] Fig. 1a to 1c show a traffic scene / scenario at a four-way intersection with an ego vehicle and two other vehicles, whereby the scenario consists of a temporal sequence of three traffic situations.

[0042] Fig. 2 illustrates the concept of a partial situation as it is used in

[0043] used within the framework of the method according to the invention for generating test situations of test scenarios.

[0044] Fig. 3 illustrates the concept of the inventive

[0045] Validation procedure using a block diagram.

[0046] Fig. 4 illustrates the generation of test scenarios within the scope of the inventive validation method using a block diagram. Fig. 5 illustrates the iterative process for generating an optimized set of test scenarios.

[0047] Description of implementation examples

[0048] The traffic scene shown in Figs. 1 a to 1 c has already been explained in detail in the introduction to the description.

[0049] Essential for the validation procedure in question is a database with predefined, formalized partial situations and associated evaluation models. Each evaluation model defines boundary conditions for the permissible behavioral options of an ego vehicle in the respective partial situation. According to the invention, the database is used to compose or construct more or less complex test situations for the validation of a behavior planner using the predefined partial situations. The boundary conditions for permissible behavioral options of the ego vehicle in these test situations are then determined using the evaluation models of the contributing partial situations, namely as a combination of boundary conditions of the individual partial situations.

[0050] Fig. 2 illustrates the concept of a partial situation. This is a situation class that is abstractly described by certain properties or elements. The description of the partial situation 20 presented here essentially comprises three elements: the ego vehicle 21, an abstract traffic infrastructure element 22 and another road user 23. Each of these three elements 21, 22 and 23 is characterized by models, rules, state parameters and / or state variables. Thus, the ego vehicle 21 in the exemplary embodiment presented here is characterized by its driving intention, dynamic state variables such as speed, acceleration and orientation, system parameters such as technical vehicle characteristics and a behavior model such as defensive driving or sporty driving.The abstract traffic infrastructure element 22 could, for example, be an intersection, a junction, or a road section. It is characterized here by a zone graph according to the SOCA method and by the traffic rules applicable to this traffic infrastructure element, e.g., “right before left.” The other road user 23 is also characterized here by its “movement” intention and by dynamic state variables, as well as by a behavior model with which assumptions can be made about the behavior of the other road user. The elements 21, 22, and 23 are embedded in a general situation context 24, into which both general context parameters 25 and general behavior models 26 of road users flow. Examples of general context parameters 25 include road condition parameters, weather parameters, and lighting parameters.The general behavior models 26 could include motion models for classes of road users, such as pedestrians, cyclists, motorcyclists, car drivers, etc., and could possibly model interactions between road users. The behavior models 26 can be modeled, for example, by continuous motion models in the form of differential equations, by Markov decision processes, or by discrete behavior spaces (discrete decision space).

[0051] The partial situation 20 is defined by all of these elements 21 to 26. Each of these elements 21 to 26 can restrict the permissible behavior of the ego vehicle. To be considered in behavior planning, all of these sources of influence must be explicitly described and / or mapped to formal parameters or decision alternatives.

[0052] A partial situation 20 defined in this way and the associated evaluation model 27 can be parameterized and instantiated by parameterization. Within the scope of the validation method according to the invention, test scenarios are generated as a sequence of test situations, which are assembled or constructed by combining, instantiating, and appropriately parameterizing selected partial situations or associated evaluation models.

[0053] In the case of the example described here, the partial situation

[0054] 20 were analyzed using the SOCA method. This resulted in an evaluation model

[0055] 27 is generated, which can be used as the basis for the analysis of all instances of partial situation 20. The evaluation model 27 is available in the form of a Zwicky box or a set of Zwicky boxes. The correspondingly data-based evaluation model 27 then provides boundary conditions for permissible behavior options of the ego vehicle in the respective instance of partial situation 20.

[0056] The evaluation models of the partial situations are preferably provided in a declarative program representation, which enables the determination of formally explorable boundary conditions for permissible behavior options of the vehicle in the respective partial situation. While Zwicky boxes are a declarative representation, they cannot be automatically data-encoded, and the boundary conditions for permissible behavior options of the ego vehicle in the respective instance of the partial situation 20, which are determined using Zwicky boxes, are initially not available in a formally explorable form. Therefore, the Zwicky box 27 of the partial situation 20 in the example presented here is transformed into a suitable declarative program representation 28, with which boundary conditions for permissible behavior options of the vehicle can be determined in a formally explorable form.It is essential that the semantics of the combinations of evaluation models are preserved during the transformation of the evaluation models, i.e., that the semantics of the combined evaluation models are representation-independent. In the embodiment described here, the Zwicky box 27 is transformed into an answer set program representation 28 using a transformation tool 21.

[0057] The validation method according to the invention essentially consists of three process steps, which are shown in Fig. 3:

[0058] In a first process step, a database 31 with predefined partial situations and at least one evaluation model for each partial situation is provided. It is essential that the partial situations of the database 31 can be combined to form different overall situations, and that boundary conditions for permissible behavior options of the ego vehicle in such an overall situation can be determined based on the evaluation models of the contributing partial situations. In a second process step, test scenarios 32 are generated in the form of a sequence of test situations. It is essential that each test situation is formed as a composition of a selection of partial situations from the database 31, which is indicated here by the arrow 301.

[0059] In a third process step—the actual validation—the behavior planner 30 to be validated processes these test scenarios 32 and produces planning results for the test situations of the individual test scenarios 32. The planning results are then analyzed using the evaluation models of the contributing partial situations. In particular, it is checked whether the planning results comply with the boundary conditions for permissible behavior options in the respective test situations. The transmission of test scenarios 32 to the behavior planner 30 to be validated and the analysis of the planning results based on the combined evaluation models is symbolized here by the double arrow 302.

[0060] At this point, it should be expressly pointed out once again that the database 31 can be specifically designed and configured to generate test situations and test scenarios, i.e. completely independently of the behavior planners 30 to be validated.

[0061] In the exemplary embodiment presented here, however, the method according to the invention is used to validate a behavior planner whose planning is based on such a database 31 with predefined partial situations and associated evaluation models. In this special case, the validation method according to the invention can use the database of the behavior planner to be validated to generate test scenarios, which is indicated here by the arrow 300.

[0062] The block diagram of Fig. 4 once again illustrates the first and second method steps of the validation method according to the invention. It shows a first database 41 with predefined partial situations and associated evaluation models. Both the partial situations and the evaluation models are available in an explorable and semantically composable formal representation. In a preferred embodiment, this is an ASP representation. The database 41 is used to generate test scenarios for the validation of a behavior planner, which was generally designed for a specific application and / or ODD. Accordingly, scenario specifications 40 are specified for the test scenarios to be generated. These are declarative specifications of event chains in the same declarative program representation as the partial situations.Usually, the number of individual situations in a test scenario—three in the example presented here—and the states that characterize the individual test situations of the scenario are specified, while the operations necessary to reach these states are not specified. Thus, the scenario specification determines which partial situations of the database 41 contribute to the individual test situations of a test scenario. The corresponding partial situations are selected from the database 41 and composed into test situation classes 421, 422, and 423 for the individual test situations of the test scenario. In addition to the selection, the scenario specification also specifies the instantiation of the partial situations and, if necessary, at least partial parameterization of the partial situations or test situation classes 421, 422, and 423.

[0063] Fig. 4 shows a second database 43 with predefined boundary conditions for the test scenarios to be generated. These can be generally valid boundary conditions for the development of a traffic scene, for example, physical laws, or ODD boundary conditions, so-called ODD filters, which define different ODDs for different behavior planners. It is essential here that the predefined boundary conditions are present in the same declarative program representation as the partial situations and the scenario specification. The predefined boundary conditions can be optionally retrieved from the database 43 to be considered when generating the test scenarios. Such a selection of predefined boundary conditions is designated here by 431.

[0064] With the help of a solver tool 44 of the common declarative program representation, all possible "reasonable" test scenarios for the given scenario specification can then be found. For example, it may be sensible for all vehicles not to move backward through their respective zones in all situations. This type of boundary condition can also be predefined in the database 43, for example, using temporal logic such as LTL (linear temporal logic). Fig. 4 shows two exemplary test scenarios 451, 452 that have been fully parameterized and data-encoded using the solver tool 44.

[0065] In most use cases, scenario specifications are easy to formulate. In contrast, it is often difficult to find a good set of test scenarios for a specific validation task. Such a set of test scenarios should contain a manageable number of test scenarios with good test coverage. If too many test scenarios are generated for a given scenario specification, the number of test scenarios used for validation must be further restricted by additional constraints defined by the user. This requires an iterative query / refinement cycle in which user input and automated generation of sets of test scenarios alternate until the user is satisfied with the query result or until the generated set of test scenarios meets a specified validation criterion.Fig. 5 illustrates such an iterative process for generating an optimized set of test scenarios. It depicts a user 50 who, for generating test scenarios, has access to a first database 41 with predefined partial situations and associated evaluation models, and a second database 43 with predefined boundary conditions for the test scenarios to be generated. In a first iteration step, the user 50 specifies a scenario specification 40 and boundary conditions 431 selected from the database 43. With the aid of a solver tool 44, a first set 55 of test scenarios is generated, as described in connection with Fig. 4. The user 50 then checks whether this first set 55 of test scenarios meets a specified validation criterion. This check is symbolized by the arrow 56.If the validation criterion is not met, the user 50 varies the scenario specification 40 and / or the selection of additional boundary conditions 431 in a further iteration step, thus modifying the model underlying the test scenarios to be generated. This user input is symbolized here by the arrow 57. Based on this user input, another set 56 of test scenarios is then generated using the solver tool 44 and tested by the user. This procedure is repeated until an optimized set of test scenarios is found that meets the validation criterion.

[0066] In another use case, a set of test scenarios, a so-called test scenario suite, is created to cover an entire ODD. To achieve this, the test scenario suite must meet a specified coverage criterion. This is usually a specified number of test scenarios per test scenario class. The appropriate coverage criterion can be developed iteratively by starting from coverage criteria for individual situations / sub-situations.

[0067] The figures together with the description of the figures illustrate the essential components and aspects that are required for the generation of test scenarios within the framework of the validation method according to the invention and with the aid of a validation system according to the invention: A database with predefined partial situations and associated evaluation models for generating test scenario models or classes, each of which consists of a sequence of test situation classes, which in turn are composed of selected partial situations of the database, wherein the partial situations and subsequently also the test scenario models are made available in a declarative program representation, a series of boundary conditions for defining and restricting the space orthe set of test scenarios of a test scenario model, whereby these constraints are also available in the declarative program representation, a validation criterion and / or a coverage criterion also in the declarative program representation and a solver tool that reads the above-mentioned artifacts and outputs a set of abstract scenarios that satisfy the validation criterion and / or the coverage criterion.

[0068] ASP is particularly suitable for declarative program representation. Alternatively, other verification tools such as model checkers, satisfiability modulo theory (SMT) solvers, binary decision diagrams, etc. can also be used. Answer set programming has the advantage of more easily supporting the creation of different numbers of instances of partial situations.

[0069] In the ASP syntax, a scenario specification for a test scenario consisting of a sequence of three test situations (three time steps) at a four-way intersection (intersection raite benz) with an ego vehicle (ego vehicle) and two other vehicles (carl and car2) can, for example, have the following form:

[0070] % There are two instances of a partial situation

[0071] % subj ect in intersection subj ect in intersection ( carl , raite benz ) . subj ect in intersection ( car2 , raite benz ) .

[0072] % There are three time steps t ime ( 1 ; 2 ; 3 ) .

[0073] % Selected constraints at the different points in time

[0074] % The ego vehicle is in its approach zone in step 1 ego zone (approach, 1).

[0075] % The ego vehicle is in the conflict zone in step 3 ego zone (conflict, 3).

[0076] % carl and car2 are in the conflict zone in step 1 subject zone ( conf lict , 1 , carl , raite benz ) . subject zone ( conf lict , 1 , car2 , raise benz ) .

[0077] Accordingly, the ego vehicle should be in its “approach zone” in the first test situation, i.e. in the first journal, and in the “conflict zone” in the third test situation, i.e. in the third journal.

[0078] The states of the two other vehicles (carl and car2) are only specified in such a way that the two vehicles in the first journal should both be in the “conflict zone”.

[0079] The ASP Solver tool fills the existing gaps in the scenario specification and outputs a number of scenarios, for example in the following form: subject in intersection ( carl , raite benz ) . subject in intersection ( car2 , raite benz ) . ego zone ( approach, 1 ) . ego zone ( conf lict , 2 ) . ego zone ( conf lict , 3 ) . subject zone (conf lict, 1, carl, raite benz), subject zone (conf lict, 1, car2, raite benz), subject zone (conf lict, 2, carl, raite benz), subject zone (conf lict, 2, car2, raite Benz ) , subject zone ( exit , 3 , carl , raise benz ) , subject zone ( conflict , 3 , car2 , raise benz ) .

[0080] This represents the scenario in which the ego vehicle enters the conflict zone between time steps 1 and 2 and is still in the conflict zone in journal three. Carl leaves the conflict zone between journals 2 and 3 and advances to the exit zone, while car2 remains in the conflict zone in all three journals. Note that in this example, there are many different possibilities for the order of movement and the actual moving vehicles. For example, car2 could move instead of carl, or both vehicles could move, or the ego vehicle could move forward after one of the other vehicles has moved on. The ASP solver tool can automatically enumerate such options for such a model.

[0081] The ASP solver tool can also be provided with coverage criteria to exclude answers, i.e. scenarios, that are similar to those already seen. For example, one could specify as a coverage criterion that one wants to see every combination of situations within a time step at least once, but it does not matter whether this is the case at time 1, 2 or 3. Furthermore, a branch coverage or condition coverage metric can be applied, which additionally take into account all possible transitions between situations. To control scenario generation, for example, a 2-fold or, more generally, a k-fold coverage can be required, where every combination of 2 (or k) different dimension values ​​should be covered for a specific time. Since, for example,Since the two states subject zone (exit, 3, carl, raite benz) and subject zone (conf lict, 3, car2, raite benz) do not influence each other, further combinations of these two dimension values ​​are unnecessary. Advantageously, a large number of such coverage metrics can be coded in an answer set program to effectively limit the number of generated test scenarios.

Claims

Claims 1 . Computer-implemented method for validating a behavior planner (30) for an at least partially automated vehicle, characterized in that • that a database (31) with predefined partial situations (20) and at least one evaluation model (27) is provided for each partial situation (20), wherein the partial situations (20) of the database (31) can be combined to form different overall situations and boundary conditions for permissible behavior options of the vehicle in such an overall situation can be determined on the basis of the evaluation models (27) of the contributing partial situations (20), • that test scenarios (32) are generated in the form of a sequence of test situations, each test situation being formed in the form of a composition of partial situations (20) of the database (31), and • that the results of the behavior planner (30) for the test situations of the individual test scenarios (32) are analyzed with the help of the evaluation models (27) of the contributing partial situations (20).

2. Method according to claim 1, characterized in that each partial situation (20) is defined as a situation class which is at least partially determined by at least one of the following elements: • an ego vehicle (21) with behavior model, • at least one transport infrastructure element (22), • a general situational context (24) and • at least one further road user (23) with a behavior model such that each partial situation (20) together with the associated evaluation model (27) can be parameterized in a situation-specific manner and / or can be instantiated by data input with situation-specific information.

3. Method according to one of claims 1 or 2, characterized in that at least part of the evaluation models (27) is based on a decomposition of the respective partial situation (20) into zone graphs and a morphological behavior analysis of the road users involved (SOCA method) and is generated in the form of Zwicky boxes (27).

4. Method according to one of claims 1 to 3, characterized in that a test scenario class is determined on the basis of a predetermined scenario specification (40) by selecting and combining corresponding partial situations from the database (41) and by instantiating and / or at least partially parameterizing the test situation classes thus generated.

5. The method according to claim 4, characterized in that individual test scenarios (451, 452) of a test scenario class are generated by completely parameterizing the test situation classes (421, 422, 423) of this test scenario class.

6. The method according to claim 5, characterized in that when generating individual test scenarios (451, 452) of a test scenario class, predefined boundary conditions (431) for the test scenarios are taken into account, in particular general boundary conditions for the development of a traffic scene and / or ODD boundary conditions which define the operational design domain (ODD) of the behavior planner.

7. Method according to one of claims 5 or 6, characterized in that a number and / or selection of test scenarios fulfilling a predetermined coverage criterion is generated per test scenario class.

8. Method according to one of claims 4 to 7, characterized in that the partial situations (20) of the database, the scenario specification (40), the general boundary conditions (431) and the ODD boundary conditions are made available in a declarative program representation, in particular in an Answer Set Program (ASP) representation, and that the test scenarios (451, 452) are automatically generated using a solver tool (44) of this declarative program representation.

9. The method according to one of claims 4 to 8, characterized in that the validation of the behavior planner is based on an optimized set (55, 56) of test scenarios which satisfies a predetermined validation criterion, and in that this optimized set (55, 56) of test scenarios is generated iteratively by varying the scenario specification (40) and / or the selection of the predefined general boundary conditions (431) and / or the ODD boundary conditions and / or the coverage criterion in the individual iteration steps until the resulting set (55, 56) of test scenarios satisfies the validation criterion.

10. Method according to one of claims 1 to 9 for validating a behavior planner (30), which has at its disposal a database (31) with predefined partial situations (20) and at least one evaluation model (27) for each partial situation (20), in order to break down a given situation into partial situations (20) of the database (31) and to determine boundary conditions for permissible behavior options of the vehicle in the given situation on the basis of the associated evaluation models (27), characterized in that the database (31) of the behavior planner (30) is used to generate the test scenarios.

11. Computer-implemented system for validating a behavior planner (30) for an at least partially automated vehicle, at least comprising: • a database (31) with predefined partial situations (20) and at least one evaluation model (27) for each partial situation (20), wherein the partial situations (20) of the database (31) can be combined to form different overall situations and boundary conditions for permissible behavior options of the vehicle in such a The overall situation can be determined on the basis of the evaluation models (27) of the contributing partial situations (20), • an interface for specifying scenario specifications (40), • a composition tool for generating test scenario classes with test situation classes (421, 422, 423) based on scenario specifications (40), • a solver tool (44) for generating individual test scenarios (451, 452) of a test scenario class, wherein a test scenario (451, 452) is formed in the form of a sequence of test situations, and • an analysis tool for analyzing the results of the behavior planner (30) for the test situations of the individual test scenarios (451, 452) using the evaluation models (27) of the contributing partial situations (20).

12. System according to claim 11, characterized in that a database (43) is provided in which predefined boundary conditions (431) for the test scenarios are stored, in particular general boundary conditions for the development of a traffic scene and / or ODD boundary conditions which define the Operational Design Domain (ODD) of the behavior planner (30), so that these boundary conditions (431) can be selectively retrieved and taken into account when generating the test scenarios (451, 452).

13. System according to one of claims 11 or 12, further comprising: a storage medium for storing test scenario classes with associated evaluation models and / or generated test scenarios.

Images