System and method for monitoring proper behavior of an autonomous vehicle

Abstract

To provide a system for monitoring a proper behavior of an autonomous vehicle and a method thereof.SOLUTION: This method includes the steps of: generating a plurality of agents, each of which describes a physical object, where at least one of the plurality of agents is an agent for a DUT; generating a plurality of scenarios, each of which models a behavior of at least one of the plurality of agents; and monitoring an interaction between the plurality of agents and the agent for the DUT as to a scenario modeling the respective agents.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This application claims the benefit of U.S. Provisional Application No. 62 / 949,098, filed December 17, 2019, the contents of which are incorporated herein by reference.

[0002] Copyright Statement All of the material in this patent document is subject to copyright protection under the copyright laws of the United States and other countries. The copyright owner has no objection to the reproduction of the patent document or patent disclosure in the official government records, but all other copyrights are otherwise reserved.

[0003] The present invention relates generally to autonomous vehicles, and more particularly to monitoring the proper performance of such vehicles. [Background technology]

[0004] Progress in the field of autonomous vehicles is rapid. Over the next decade, we should see an increasing number of such vehicles on the roads, and experimental vehicles are roaming the roads of many cities around the world. Like all advanced devices designed by humans, autonomous vehicles not only enjoy the benefits of human ingenuity, but also experience its shortcomings. Shortcomings manifest themselves as undesirable, unexpected, or erroneous behavior of the autonomous vehicle, endangering the vehicle's occupants as well as other people, animals, and property around the vehicle.

[0005] To prevent such errors from occurring, vehicles are first tested before going on the road, and then, when they are on the road, additional precautions are put in place to ensure accidents are prevented. In addition, each such vehicle is assigned a driver who has the ability to override the vehicle's operation when an error in operation or response occurs. This, of course, makes it possible to capture such sequences and update the vehicle's control system, so that future instances of such dangerous situations will be prevented. However, these solutions are error-prone because they rely heavily on capturing such errors as a result of operator intervention or instances where some kind of damage has occurred. Errors that lead to undesirable consequences are not effectively monitored or captured when undesirable consequences cannot occur.

[0006] It is therefore desirable to provide a solution that allows for monitoring the operation of an autonomous vehicle based on predetermined expectations of correct operation, rather than waiting for a drastic error to occur. It would therefore be advantageous to systematically monitor the performance of an autonomous vehicle while testing it by exposing it to multiple scenarios in a controlled environment, such as a simulation or test track. Summary of the Invention

[0007] The following is a summary of some exemplary embodiments of the present disclosure. This summary is provided for the reader's convenience to provide a basic understanding of such embodiments, but does not define the breadth and scope of the present disclosure as a whole. This summary is not an extensive overview of all contemplated embodiments, and is not intended to identify key or critical elements of all embodiments or to delineate the scope of some or all aspects. The sole purpose of this summary is to present some concepts of one or more embodiments in a simplified form as a prelude to the more detailed description that is presented later. For convenience, the terms "some embodiments" or "certain embodiments" may be used herein to refer to a single embodiment or to multiple embodiments of the present disclosure.

[0008] Certain embodiments disclosed herein include: The method includes a computer implemented for identifying at least an occurrence of a scenario, the method including the steps of receiving one or more scenarios, each scenario referencing at least a first object and a second object, each of the first object and the second object being described by the temporal behavior of a first plurality of agents, receiving a data stream of behavior of the plurality of objects, the plurality of objects being described by the temporal behavior of a second plurality of agents, identifying an occurrence of each of the one or more scenarios in the received data stream based on the temporal behavior of the first plurality of agents and the temporal behavior of the second plurality of agents, and generating a notification for each identified scenario in the received data stream. Includes.

[0009] Certain embodiments disclosed herein are Also , A computer implemented method for identifying at least a scenario occurrence includes a non-transitory computer-readable medium having stored thereon instructions to cause a processing circuit to perform the following steps: receiving one or more scenarios, each scenario referencing at least a first object and a second object, each of the first object and the second object being described by the temporal behavior of a first plurality of agents; receiving a data stream of behavior of the plurality of objects, the plurality of objects being described by the temporal behavior of a second plurality of agents; identifying an occurrence of each of the one or more scenarios in the received data stream based on the temporal behavior of the first plurality of agents and the temporal behavior of the second plurality of agents; and generating a notification of each identified scenario in the received data stream. Includes.

[0010] Additionally, certain embodiments disclosed herein may include: Identifying the occurrence of one or more scenarios involving the object The system includes a system for: a processing circuit communicatively connected to the network interface, an input / output (I / O) interface, and a processing unit configured to execute a plurality of instructions provided thereto; and a memory having instructions therein for execution, wherein when the processing unit executes the instructions, the monitoring system performs the following operations: receive one or more scenarios, each scenario referencing at least a first object and a second object, each of the first object and the second object being described by the temporal behavior of a first plurality of agents; receive a data stream of behavior of a plurality of objects, each of the plurality of objects being described by the temporal behavior of a second plurality of agents; identify each occurrence of the one or more scenarios in the received data stream based on the temporal behavior of the first plurality of agents and the temporal behavior of the second plurality of agents; and generate a notification of each identified scenario in the received data stream. It is configured as follows.

[0011] The subject matter disclosed herein is particularly pointed out and distinctly claimed in the claims at the conclusion of this specification. The above and other objects, features, and advantages of the disclosed embodiments will become apparent from a consideration of the following detailed description in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic diagram of a monitoring system for activating agents and scenarios to monitor the behavior of an autonomous vehicle, according to one embodiment. [Figure 2] 1 is a flow diagram illustrating a method for deploying a monitoring system for an autonomous vehicle, according to one embodiment. [Figure 3] 1 is a flow diagram illustrating a method for generating at least one agent of a monitoring system, according to one embodiment. [Figure 4A] FIG. 1 is a schematic illustration of an interrupt and slow down scenario from start to finish, according to one embodiment. [Figure 4B] FIG. 1 is a schematic illustration of an overtaking process in a cut-in and slow-down scenario according to one embodiment; [Figure 4C] FIG. 1 is a schematic illustration of an interrupt process for an interrupt and slow down scenario, according to one embodiment. [Figure 4D] FIG. 1 is a schematic illustration of a deceleration process for an interrupt deceleration scenario, according to one embodiment. [Figure 5] FIG. 1 is a schematic illustration of coverage metrics for an interrupt and slowdown scenario, according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] It is important to note that the embodiments disclosed herein are merely illustrative of the many advantageous applications of the innovative teachings herein. In general, statements in the specification of this application do not necessarily limit the scope of any of the various claims. Moreover, some statements may apply to some inventive features and not to other features. In general, unless otherwise indicated, singular elements may be in the plural and vice versa without loss of generality.

[0014] The field of autonomous vehicles requires vehicles to operate with a level of perfection that exceeds that of humans. However, errors and faults in the designs and programs designed and programmed by humans and installed in vehicles can lead to undesirable and unpredictable results. Therefore, the measurable scenario description language (MSDL) is used to generate agents that operate monitoring devices that monitor data streams. Such data streams provide information about the behavior of the monitored vehicle compared to the behavior described by the MSDL scenarios. Agents are generated using MSDL and executed to monitor the incoming streams, detect and report anomalies, key performance indicators, and scenario coverage. That is, if the behavior of the monitored vehicle differs from the expected value, the anomaly is reported. MSDL can also describe unmonitored elements in the monitored vehicle's environment.

[0015] 1 depicts an example schematic diagram of a monitoring system 100 for activating agents and scenarios to monitor autonomous vehicle behavior, according to one embodiment. Monitoring system 100 includes a processing unit 110 communicatively coupled to a memory 120. Memory 120 may include both volatile memory, such as random-access memory (RAM), and non-volatile memory, such as read-only memory (ROM) and flash memory. As described further herein, memory 120 may have a portion of memory 120 allocated to include instructions that can be executed by processing unit 110.

[0016] Database (DB) 130 is further coupled to processing unit 110 and may contain various types of data, as discussed further herein. Database (DB) 130 may contain instructions to be executed by processing unit 110 or data to be processed by processing unit 110. Database (DB) 130 may also accept data to be prepared or processed by processing unit 110. Data contained in database (DB) 130 may include pre-prepared entities, such as agents, which are discussed in more detail herein. Additionally, database (DB) 130 may contain data streams, such as video clips, which may be used by the disclosed embodiments to monitor the behavior of an autonomous vehicle.

[0017] A network interface 140 is further connected to the processing circuit 110. The network interface 140 allows the monitoring system 100 to send and receive data over a network, which may be wired or wireless. Relevant types of networks include, but are not limited to, local area networks (LANs), wide area networks (WANs), metro area networks (MANs), cellular networks, Wi-Fi networks, and the like, as well as any combination thereof. The data streams described herein may be provided to the monitoring system 100 using the network interface 140, in one embodiment. Additionally, an input / output (I / O) interface 150 may be connected to the processing unit 110. Such interfaces may provide connectivity to various devices, including, but not limited to, computer screens, touchscreens, keyboards, mice, and other similar input, output, or input / output devices. The uses of the various components of the monitoring system 100 are described in more detail herein.

[0018] The processing circuitry 110 may be implemented as one or more hardware logic elements and circuits. For example, without limitation, illustrative types of hardware logic elements that may be used include a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-a-chip system (SOC), a graphics processing unit (GPU), a tensor processing unit (TPU), a general-purpose microprocessor, a microcontroller, a digital signal processor (DSP), and the like, or other hardware logic elements capable of performing calculations or other information manipulations.

[0019] The memory 120 may be volatile (eg, RAM), non-volatile (eg, ROM, flash memory), or a combination thereof.

[0020] In one configuration, software for implementing one or more embodiments disclosed herein may be stored in database 130. In another configuration, memory 120 is configured to store software code, such as software code 125. Software code 125 includes any instructions developed using MSDL. Software code 125 should be broadly interpreted to mean any type of instructions, whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise. Instructions may include code (e.g., code in source code format, binary code format, executable code format, or any other suitable format). Processing unit 110 executes instructions to perform the various processes described herein.

[0021] Database (DB) 130 may be any type of storage device, such as magnetic storage, optical storage, and the like, and may be implemented as, for example, flash memory or other memory technology, CD-ROM, Digital Versatile Disk (DVD), or other medium that can be used to store the desired information.

[0022] It should be understood that the embodiments described herein are not limited to the particular structure shown in FIG. 1, and that other structures may similarly be used without departing from the scope of the disclosed embodiments.

[0023] An example scenario will be described to illustrate the operation of the surveillance system 100. It will be appreciated that other similar scenarios can be easily developed and deployed to run on the surveillance system 100 operating as a surveillance device. For the sake of quick illustration, a scenario known as a "cut in and slow down" scenario will be used. This is a scenario frequently encountered in traffic where one vehicle cuts in front of another vehicle (e.g., an autonomous vehicle) and slows down.

[0024] Below is a step-by-step description using MSDL of how a "cut in and slow down" scenario is captured according to a disclosed embodiment, from the definition and setup of the scenario, to the logical description of the scenario, and the final realization of the scenario. MSDL is used to ensure that an autonomous vehicle is behaving as expected when testing or monitoring the autonomous vehicle's response, either in simulation or from actual data streams captured when the autonomous vehicle is on the road. The onus for validation is on the programmer who uses MSDL to define the scenario from these elements using the monitoring system 100 as described herein.

[0025] The definition of a scenario for verifying or monitoring a desired behavior is described in an instant exemplary case. It will be understood that other scenarios are possible without departing from the scope of the present invention. In this case, an autonomous vehicle may proceed at an essentially constant speed, staying in its lane as much as possible. Another vehicle may approach from behind the autonomous vehicle in a different lane and overtake the autonomous vehicle. This vehicle may then cut in front of the autonomous vehicle in its lane.

[0026] If this vehicle then slows down in front of the autonomous vehicle in the autonomous vehicle's lane, the autonomous vehicle may be forced to take necessary measures to maintain a safe distance from this vehicle. It will be understood that the given example includes two behaviors: the behavior of the autonomous vehicle and the behavior of the other vehicle. The scenario definition also defines coverage metrics and a grading mechanism to prove that validation goals have been achieved. Metrics may target operational parameters such as, but not limited to, following distance and speed.

[0027] According to an embodiment, all scenarios are defined using MSDL as described herein, and the execution of the scenarios is monitored, for example, using system 100. Metrics are also referred to in the art as key performance indicators (KPIs). KPIs are typically, but not exclusively, physical parameters such as speed, distance, acceleration, and the like. Coverage metrics are typically, but not limited to, logical, Boolean, or discrete statements, queries, and the like, such as, for example, "Did a handover occur?", "Did an interrupt occur?", "Did an interrupt occur from the left or right?"

[0028] FIG. 2 is an example flow chart 200 illustrating a method for deploying a monitoring system 100 for an autonomous vehicle, according to one embodiment.

[0029] At S210, a plurality of agents are generated, including at least an agent for a device under test (DUT). In a typical embodiment, the DUT agent is for an autonomous vehicle (AV). Each agent is described using MSDL, and examples of usage are provided. A more detailed, non-limiting description of MSDL is given below.

[0030] An agent may use previously created agents to describe a newly created agent. Thus, a hierarchical structure of agents is established. In one embodiment, a root agent is an agent that calls all agents, either directly or through its hierarchical levels. Agents may describe DUTs, vehicles, roads, sidewalks, people, animals, traffic lights, cones, barriers, bicycles, trains, weather elements, hazards, and similar physical objects. It should be understood that the provided list is not exhaustive, and other agents may be described using MSDL, an agent hierarchy, or both. It should be further understood that such agents may be manually created via a user interface (not shown) executed by processing unit 110, or automatically created.

[0031] Automatic extraction can be achieved by using a data stream, such as, but not limited to, a video stream, and an agent is generated for the desired physical object. The data stream can be provided in real time through network interface 140 or stored in database (DB) 130 for offline use. Agents can include references to lower-level agents, allowing previously defined agents not to be recreated. For example, but not limited to, if a vehicle activates its lights as a result of a change in ambient lighting, but a physical object that turns on its headlights already exists as an agent, it may be useful to use the previously generated agent when generating an agent for the vehicle.

[0032] Multiple scenarios are generated in S220, and the scenarios may be described using MSDL and may model the behavior of agents. For example, without limitation, a car may have a driving scenario, a pedestrian may have a walking scenario, a traffic signal may have a light changing scenario, etc. It should be appreciated that traffic scenarios may include, by way of illustration and without limitation, a cut-in scenario in which a car cuts in front of the DUT, a cut-in scenario from both sides, e.g., two cars cut in front of the DUT from two separate lanes, a car cuts in front of the DUT and then slows down, a cut-in and slows down scenario, and the like.

[0033] These complex scenarios may activate lower-level scenarios in the scenarios involved to appropriately implement them. That is, an "interrupt" may invoke a driving scenario for each of the vehicles, thereby invoking the "interrupt" scenario multiple times. It will be understood that this list is not exhaustive, and other scenarios may be described using MSDL, a scenario hierarchy, or both. It will further be understood that such scenarios may be generated manually via a user interface (not shown) executed by processing unit 110, or may be generated automatically. Automatic extraction may be achieved, for example, but not limited to, by using a data stream, such as a video stream, from which the scenario is generated. The data stream may be provided in real time through network interface 140 or stored in database (DB) 130 for offline use. As further described herein, a scenario may include references to lower-level scenarios, allowing previously defined agents not to be replayed.

[0034] In S230, the generated agents and scenarios are executed by the monitoring system 100 to monitor the behavior of the agents of the DUT.

[0035] In S240, a test is performed to determine whether an error indication was provided. In one embodiment, this test may detect when a scenario has occurred successfully. The testing step may further include storing key performance indicators (parameters) and their coverage data. For example, without limitation, an error may be generated when the DUT's agent enters a low-light area but the DUT's headlights are not activated. In another case, such as a cut-in and slow-down scenario, the DUT's agent is expected to slow down the DUT at a safe distance from the cutting-in vehicle. It will be appreciated that a minimum distance between vehicles is a non-limiting example of a key performance indicator (KPI) that may be stored once a cut-in and slow-down scenario is detected. If an error is detected in S240, execution continues at S250. Otherwise, execution continues at S260.

[0036] At S250, a notification is generated indicating that an error has been detected. In one embodiment, the scenario that led to the error notification may also be provided, and other notifications may be provided that allow the DUT's agent in a particular instance to recreate the sequence that led to the error, as well as any combination thereof.

[0037] In S260 it is checked whether monitoring should continue, for example by creating additional agents and / or scenarios, and if so execution continues in S230, otherwise execution terminates.

[0038] FIG. 3 depicts an example flow diagram 300 illustrating a method for creating at least one agent of a monitoring system, according to one embodiment.

[0039] At S310, a data stream is received by the surveillance system 100. The data stream may be received through a network interface 140, for example, for real-time video, or from a database (DB) 130, for example, for offline video processing.

[0040] In S320, a physical object is selected, such as a DUT, a vehicle, a sidewalk, a person, or the like. The selection may be performed by a user interface (not shown), for example, that enables a pointing device (not shown) connected to I / O network interface 140 to be used to point to and select the physical objects provided by the data stream. It will be understood that the selection in S320 may be extended to select multiple physical objects for generating respective agents for each of the physical objects without departing from the scope of the disclosed embodiments.

[0041] In S330, an agent for the physical object is generated, and the physical object may further use previously defined agents, thereby generating a hierarchical structure of agents. The agent generated in S330 may be of a specific type corresponding to the characteristics of the physical object, and the object type is defined using MSDL. In one embodiment, manual user intervention may direct the generation of agents when and where necessary. Furthermore, two or more agents may be generated in parallel.

[0042] In S340, the generated agent is stored in a memory, such as memory 120 in one embodiment or database (DB) 130 in another embodiment. The agent is generated using MSDL as discussed herein.

[0043] At S350, it is checked whether more agents are to be created, for example based on the data stream received by the monitoring system 100, and if so, execution continues at S320; otherwise, execution terminates.

[0044] 4A is an example schematic diagram 400A of a cutting-in and slowing down scenario, according to one embodiment. An autonomous vehicle (AV) 410 is referenced in the diagram as EGO, and another vehicle 420 is referenced in the diagram as CAR. Thus, the autonomous vehicle 420 begins the scenario at a starting point 410S, and the other vehicle 420 begins the scenario at a position 420S that is later than position 410S. The other vehicle 420 has a relative speed greater than zero with respect to the AV 410, overtakes the AV 410 on travel path 440, cuts into the AV 410's lane, and then slows down to reach position 420E. This forces the AV 410, traveling on path 450, to slow down to accommodate the speed change of the other vehicle 420 in order to maintain safety requirements, thereby terminating the scenario at position 420E.

[0045] The illustration provided in FIG. 4A can be implemented in three separate ways: overtaking, cutting in, and slowing down the AV 410 (FIGS. 4B-4D).

[0046] FIG. 4B is a schematic illustration 400B of an overtaking process for a cut-in and decelerate scenario, according to one embodiment. The first process illustrated in FIG. 4B is defined between a starting position and a position where vehicle 420 exits ahead of AV 410. In this process, vehicle 420 accelerates from 420S to 420A over distance 440A to overtake AV 410A, which remains in the same lane. During this period, AV 410 maintains its speed and trajectory for distance 450A, starting at 410S and continuing to 420A.

[0047] FIG. 4C is a schematic illustration 400C of a cut-in process for a cut-in and decelerate scenario, according to one embodiment. In the second process depicted in FIG. 4C, a vehicle starting at 420A changes lanes, moves in front of AV 410A at location 420C, and travels distance 440C. As it travels distance 450C from AV 410A to reach AV 410C, it continues to maintain its speed and trajectory unless the cut-in is aggressive (i.e., a sharp lane change and / or a short distance between AV 410A and vehicle 420A). The speed of vehicle 420A at the end of the second process may be equal to or greater than its speed at the start of the second process. It will be appreciated that the new position of AV 410 (AV 410 after being cut in by vehicle 420A) is now shown as 410C, and vehicle 420A is now shown as 420C (vehicle 420 after cutting in front of AV 410A (AV 410A is now in a different position and is therefore referred to as AV 410C)).

[0048] FIG. 4D is a schematic illustration 400D of the deceleration process for an interrupt deceleration scenario, according to one embodiment. In the third process shown in FIG. 4D, vehicle 420C brakes and AV 410C must respond in a similar manner when it is at a distance 450E from AV 410C. Entering this third process, AV 410C's final speed is lower than the starting speed of either vehicle. When this scenario ends, AV 410 is at the position of AV 410E, and AV 420 is at the position of AV 420E. According to one embodiment, MSDL allows for the segmentation of a scenario into smaller sub-scenarios or sub-processes, which may result in a built-in chaining mechanism, for example, via the do_serial command, as described further herein.

[0049] 5 is a schematic illustration of coverage metrics for a cut-in and slow-down scenario, according to one embodiment. The notation described in FIG. 5 is: rel_d_cls describes the relative distance at the lane change initiation point ([0..1500] cm in 50 cm intervals), ego_v_cls describes the absolute speed of AV 410 at the lane change initiation point ([10..130] km / h in 10 km / h intervals), rel_v_cls describes the relative speed of vehicle 420 with respect to AV 410 at the lane change initiation point ([-33..33] m / s in 1 m / s intervals), rel_d_cle describes the relative distance between AV 410 and vehicle 420 at the lane change end point ([0..3000] cm in 50 cm intervals), and ego_v_cle describes the absolute speed of AV 410 at the lane change end point. rel_v_cle describes the absolute speed of the vehicle 420 with respect to the AV 410 at the lane change end point ([10..130] km / h in 10 km / h intervals), and rel_v_cle describes the relative speed of the vehicle 420 with respect to the AV 410 at the lane change end point ([-33..33] m / s in 1 m / s intervals).

[0050] Coverage metrics such as those described above and the like, and any combination thereof, may be used to manually or automatically implement scenarios according to the disclosed embodiments.

[0051] An example of an interrupt and slow down scenario can be defined using MSDL as follows:

number

[0052] In the above example, we define the field car1 for the car agent's type, the initial side the car starts from relative to ego, and the road extent. We then add constraints that the path length is [100..250] meters and the path must have at least two lanes.

[0053] In MSDL, all fields can be randomized by default unless otherwise specified. This means that each field is given a random value at runtime within their normal value space. Every field has a physical quantity space defined by the value range of its type, and a normal value space that is the intersection between the physical quantity space and any constraints applied to it. For example, the side field has a physical quantity space of [left, right] and the same normal value space, given that no constraints are applied. On the other hand, the path field has a physical quantity space that is the set of all road ranges that the map can provide, while the normal value space is reduced to only road ranges that have a length of [100..250] meters or less and have at least two lanes. "+path_ * " are constraints that apply to the characteristics of the selected road range. In one embodiment, MSDL may provide a number of agents to assist in defining scenarios or agents. Agents and scenarios may be defined automatically as described herein or manually, as needed. Any scenario must be defined and executed in the context of an agent.

[0054] Therefore, the behavior of the scenario can be defined as follows:

number

[0055] In the above example, "do serial" is a built-in operator that describes sequential activities that are chained together in a defined order. In the exemplary embodiment, "get_ahead_of_ego" executes first, followed by "cut_in," then "slow_down." "phase" is another built-in operator that describes parallel activities. For example, "ego_car.drive()" and "car1.drive()" execute in parallel. Each phase has a start and an end point. Similarly, "do serial" begins at the start of the first phase and ends at the end of the last phase.

[0056] A complete list of process termination conditions can be found in the SDL language manual. According to one embodiment, the MSDL monitoring system 100 plans the trajectory of car1 so that car1 is at the side of ego at the start of the cut_in process. In other words, the MSDL-based monitoring system 100 is adapted to infer all necessary movements to obtain the desired position at the start of the cut_in, thereby resulting in a significant technical improvement over prior art solutions by easing the test engineer's job and generating more diverse and specific scenarios.

[0057] It will be understood that the given description describes a situation in which the system actively controls car1 to trigger a scenario. In one embodiment, car1 is selected from a data stream (e.g., a video clip) and monitored to see if the interaction described in the "interrupt and slow down" scenario occurs. In an exemplary embodiment, system 100 (FIG. 1) is passive with respect to controlling car1.

[0058] In one embodiment, coverage can be further defined within a scenario, in the scenario definition, or in an extension (also called an "aspect") of the scenario. The following are non-limiting examples of interruption coverage scenario extensions:

number

number

[0059] According to one embodiment, a test engineer defining coverage considers two aspects: which parameter values ​​to sample and when to sample them. Using the MSDL monitoring system 100, a test engineer specifies the parameters to sample and is provided with configuration to refine the set of values ​​that need to be sampled (e.g., units, value ranges). Coverage definitions use events that define the moments in time at which parameters should be sampled. In the above example, "change_lane.start" and "change_lane.end" are sampling events that activate parameter sampling. Test engineers can use predefined events or define and launch specialized events that reuse such predefined events.

[0060] According to one embodiment, the first stage of the test flow is to load the MSDL source into the monitoring system 100. The next step is to load and interpret the maps used by the scenario. At this point, the MSDL monitoring system 100 is ready to plan a list of actions. This process extends the behavior described in the scenario into a higher-granularity driving scenario. This process also chains together the various processes in the scenario.

[0061] If the planning process is successful, testing moves to simulation. During this process, the MSDL monitoring system 100 begins interacting with the simulator, advancing the simulation and obtaining agent dynamic information (position, velocity, acceleration) to adjust the plan as needed to achieve the plan's goals. Notably, in any AV simulation, traffic conditions impose a small amount of unpredictability on AVs, which can employ unpredictable decisions to deal with situations. This means that the plan will not necessarily match reality, and the MSDL monitoring system 100 mitigates side effects of AV behavior by adjusting the plan at the end of each simulation step.

[0062] In another embodiment, an MSDL source file may be loaded into a system (e.g., monitoring system 100). A data stream is then provided. Based on the selection of physical objects, agents are generated according to the embodiments described and disclosed herein. The activity of the generated agents is then monitored, e.g., in relation to an "interrupt and slow down" scenario. The monitoring system generates an event when it detects the occurrence of a step in the scenario. Coverage information is stored by the system, and respective messages are issued accordingly.

[0063] The following is a description of MSDL, which is utilized by various embodiments to define agents and scenarios. To verify the safety of an autonomous vehicle (AV) or advanced driver assistance system (ADAS), the vehicle's behavior or systems should be observed in various situations or scenarios. Using the Measurable Scenario Description Language (MSDL), scenarios can be defined and generated that describe the behavior of an AV in its environment as well as other actors. Actors include vehicles, pedestrians, weather, road conditions, etc.

[0064] Because MSDL scenarios are high-level descriptions, many specific variants of a scenario can be generated by varying scenario parameters such as speed, vehicle type, and weather conditions. In one embodiment, the MSDL tool can automatically generate these variants within a specified set of constraints. Such constraints can be provided by a user. In one embodiment, the MSDL tool collects and aggregates parameter data from successful tests, thereby enabling the safety of an AV to be measured.

[0065] MSDL is a largely declarative programming language. The only scenario that executes automatically is the top-level scenario. The execution flow of the program can be controlled by adding scenarios to the top-level scenario. MSDL is an aspect-oriented programming language; changes to the behavior or aspects of some or all instances of an object can be made to suit the purposes of a particular verification test without disturbing the original description of the object.

[0066] MSDL is a small, domain-specific language designed to describe scenarios in which actors (sometimes called agents), such as cars and pedestrians, progress through an environment. These scenarios have parameters that allow for control and constraints on the actors, their movements, and the environment. MSDL is designed to facilitate the composition of scenarios and tests, allowing users to define complex behavior using their own techniques. A minimal, extensible set of actors and scenarios contains the basic building blocks. Some built-in scenarios perform tasks common to all scenarios, such as implementing parallel execution. Other built-in scenarios describe relatively complex behavior, such as the "car.drive" scenario.

[0067] By calling these scenarios, more complex behaviors can be described, such as a vehicle approaching a give-way sign. Multiple scenarios can be mixed for even greater complexity; for example, a weather scenario can be mixed with a car scenario. New actors and new scenarios are easily created as needed, either from scratch or using already defined scenarios. For example, the scenario "cut_in" presented below is defined using the scenario "car.drive". In one embodiment, a standard scenario library is provided to support all scenarios. New or customized scenarios are added to the library as needed.

[0068] MSDL Building Blocks The building blocks of MSDL are data structures that contain at least the following: · Simple Structs - Basic entities with attributes, constraints, etc. Actors - similar to structs but also have associated scenarios. · Scenario - describes the behavior of actors.

[0069] These data structures have attributes that hold scalar values, lists, and other structures. Attribute values ​​are written as expressions or computed by external method definitions. Attribute values ​​can be controlled using keep() constraints, e.g., keep(speed < 50kph).

[0070] Attribute values ​​in a scenario can also be controlled using keep() constraints or scenario modifiers such as speed(). For example, speed(20kph, faster_than: car1).

[0071] The data structure also defines events, for example the event too_close is (distance_between(car1, car2) < 10m). The behavior of a scenario can be described by calling built-in scenarios. To execute a scenario in serial or parallel execution mode or mix it with another scenario, operator scenarios such as serial, parallel, or mixed can be activated or called. Other built-in scenarios perform time-related actions such as firing, waiting, or reporting an error.

[0072] Example Scenario Example 1 shows how to define and extend actors using MSDL. First, the actor car_group is defined with two attributes. This actor is then extended in a different file to add another attribute.

number

[0073] Example 2 shows how to define a new scenario called two_phases using MSDL. This scenario defines a single attribute car1, which is a green truck, and uses serial and parallel operators to activate the car1.drive scenario, then applies the speed() modifier. The Two_phases scenario works as follows: During the first phase, car1 accelerates from 0 kph to 10 kph. During the second phase, car1 maintains a speed of 10-15kph.

number

[0074] Example 3 shows how to define the tests to be performed as follows: 1. Import the appropriate configuration to run the test using a simulator, for example the SUMO simulator. 2. Import the defined two_phases scenario. 3. Extend the predefined, initially empty top.main scenario to launch the imported two_phases scenario.

number

number

[0075] Example 5 shows how to define a two_cut_in scenario using a cut_in scenario. The two_cut_in scenario executes a cut_in scenario from the left side followed by a cut_in from the right side. Furthermore, the colors of the two cars involved are constrained to be different.

number

[0076] Example 6 shows how to run cut_in with concrete values. The original cut_in specified a range, so each run will, by default, select a random value within that range. Tests can be defined as concrete using constraints.

number

[0077] Example 7 shows how to mix multiple scenarios: a cut_in scenario, another scenario called interceptor_at_yield, and a set_weather scenario. Because a dangerous situation is desired, the mix_dangers scenario has a single attribute of type weather_kind constrained to not be nice (i.e., !nice). This attribute is passed to set_weather.

number

[0078] Example 8 implements mix_dangers, where a specific weather condition (rain) is specified rather than a non-specific term (e.g. bad weather).

number

[0079] Overview of Vocabulary Provisions MSDL is a scripting language similar to Python. MSDL programs consist of statements that declare or extend types such as structs, actors, and scenarios, or import other files that consist of statements. Each statement contains an optional list of members, indented one unit (a consistent number of spaces) from the statement itself. Each member in a block may have its own member block, indented one unit from the member itself, depending on its type. Thus, the hierarchical structure of an MSDL program and the place of each member in that hierarchy is strictly dictated by indentation.

[0080] Normal indentation indicates members at the same level in the hierarchy. It is recommended that multiples of four spaces (blanks) be used to indicate successive hierarchical levels, but other multiples (two, three, etc.) are also acceptable as long as usage is consistent. Inconsistent indentation within a block is an error. Tabs in code are converted to spaces. Members (other than strings) that are too long to fit on a single physical line can be continued on the next line by preceding the newline character with a backslash character (\). However, lines with an open parenthesis (or open square bracket [) flow across the newline without requiring a backslash character. Inline comments are preceded by a hash tag character (#) and terminate at the end of the line. Block comments are allowed. Each line of a block must begin with the characters / * and end with the characters * / . Nested block comments are allowed. Line breaks within a comment and the indentation of comments do not affect the nesting of code.

[0081] MSDL Configuration A statement is a top-level construct that exists outside of other constructs in a program. Statements include: Enumeration declaration ·Struct Declaration Actor Declaration Scenario Declaration Scenario Modifier Declaration Extensions to those declarations Import Statements

[0082] A statement is a top-level construct that defines or extends a type or imports a file consisting of statements. Enumerated type declarations define explicitly named sets of values. For example, the enumerated type driving_style might define two sets of values: normal and aggressive. Struct declarations define composite data structures that store various types of related data. For example, a struct called car_collision might store data about vehicles involved in a collision. Actor declarations model entities such as cars, pedestrians, environmental objects like traffic lights, etc. Statements are composite data structures that store information about these entities. In contrast to structs, statements can also be associated with scenario declarations or extensions. An actor is therefore a collection of both related data and declared activities.

[0083] A scenario declaration defines a complex data structure that describes the behavior or activity of one or more actors. The behavior of a scenario can be controlled, and data about the execution of a scenario can be collected by declaring data fields and other members in the scenario itself or in the scenario's associated actors or structs. Example scenarios are car.drive, dut.cut_in, and dut.cut_in_with_person_running. A scenario modifier declaration modifies, but does not define, the behavior of a scenario by constraining attributes such as speed or position. A scenario modifier declaration may include previously defined modifiers. Extensions to an existing type or subtype of an enumeration, struct, actor, or scenario are added to the original declaration without modifying it. This capability allows extending a type for a specific test or set of tests.

[0084] Struct, actor or scenario members. The following constructs can only appear inside a struct, actor or scenario declaration or extension. These constructs include: Coverage definition ·event Field declaration Field constraints (keep()) ·External method declaration When subtype Note that scenarios are associated with specific actors, but are declared as top-level statements.

[0085] The constructs described in this paragraph may only appear inside a declaration or extension of a struct, actor, or scenario. Cover definitions allow sampling of key parameters related to scenario execution. Collecting this data across multiple executions of a scenario makes it possible to evaluate the safety of an AV. For example, if a car actor has a field speed, the value of this field may be collected at key points during the scenario.

[0086] Coverage definitions appear in scenarios to have access to the scenario's parameters and vary the coverage definition according to the scenario. Field declarations define named data fields of any scalar, struct, or actor type, or a list of any of these types. The data type of this field must be specified. For example, this field declaration defines a field named legal_speed of type speed. Field constraints defined with keep() constrain the values ​​that can be assigned to or generated for a data field. For example, the following keep() constraint keeps randomized values ​​of legal_speed below 120kph: keep(legal_speed < 120kph) This constraint can also be written as follows, with an implicit variable referencing the legal_speed field: legal_speed: speed with: keep(it < 120kph)

[0087] An event defines a specific point in time. An event is raised by an explicit firing action in a scenario or by the occurrence of another event to which it is bound. Scenarios and scenario processes have three predefined events: start, end, and fail. When subtypes extend an object, they occur when a specified condition is true. For example, if the actor my_car has a field of type driving_style, the actor's attributes or behavior may be different when the value of driving_style is aggressive than when it is normal. External method declarations, which identify required code written in other programming languages ​​such as C++, Python, and e-verification languages, can be invoked from an MSDL program. For example, an external method can be invoked that calculates and returns a value based on the scenario's parameters.

[0088] Scenario members. Scenario members appear inside a scenario declaration or extension. Scenario members include: Allowed members in a struct or actor Scenario Modifier Activation ·do(behavior definition)

[0089] Scenarios have two members that are not allowed in structs or actors. Scenario modifiers are scenarios that constrain various attributes of the scenario's behavior. Scenario modifiers do not define the primary behavior of the scenario. There are both relative and absolute modifiers. In the example below, the speed() modifier sets the speed of the affected car to be 1-5 kph faster than car1. speed([1..5]kph, faster_than: car1)

[0090] The "do" scenario member defines the behavior of the scenario when it is launched.

[0091] Scenario Activation. The following types of scenarios can be activated: Operator Scenarios Event-related scenarios Zero Hour Scenario User-defined scenarios

[0092] Scenario launches extend the execution of an MSDL program. The built-in top.main scenario is launched automatically. top.main must be extended to launch other scenarios, which define the behavior of the entire MSDL program.

[0093] Expression. Expressions can be used inside statements or members to evaluate values ​​of a specified type. Expressions are allowed as specified in the configuration. Expressions must evaluate to a specified type. Expressions can contain calls to external methods that return values.

[0094] Data Types MSDL defines the following data types: · Scalar types that hold one value at a time: numeric types, logical types, enumeration types. A list type that holds multiple values ​​of one type. A string type that holds a series of ASCII characters enclosed in double quotes. A resource type that holds a list of map junctions and segments. A real type that holds floating-point values. Composite types that hold multiple values ​​of multiple types.

[0095] physical type Physical types are used to characterize physical movement in space and include speed, distance, angle, etc. When specifying a value for one of these types in an expression or when defining coverage for it, the respective units must be defined and used. A selection of commonly used types of units may be selected, as shown in the table below. Physical constants have an implicit type. For example, 12.5km has an implicit type of distance. Examples: 2meters, 1.5s, [30..50]kph.

[0096] The following are standard physical units used: Acceleration: kphps (= kph per second), mpsps (= meters per second per second). Angle: deg, degree, rad, radian angular_speed:degree_per_second, radan_per_second. Distance: mm, millimeter, cm, centimeter, in, inch, feet, m, meter, km, kilometer, mile. Speed: kph, kilometer_per_hour, mph, mile_per_hour. Temperature: c, celsius, f, fehrenheit. Time: ms, millisecond, s, sec, second, min, minute, hr, hour. Weight: kg, kilogram, ton.

[0097] Enumerations An enumeration represents a set of explicitly named values. In the following example, the enumeration driving_style has two values: aggressive and normal. type driving_style: [aggressive, normal]

[0098] List type Lists are a way of describing containers of similar values ​​in MSDL. A list can contain any number of elements from a data type. For example, a convoy can be declared to contain a list of car actors or a shape as a list of points. A list literal is defined as a comma-separated list of items, for example: [point1, point2] The [n..n] notation is not allowed for lists as it is reserved for ranges. An example list is as follows: convoy: list of cars shape: list of point An example of a list specification is: shape: list of points with: keep(it == [map.explicit_point(”-15”,0,20m,1), map.explicit_point(”-15”,0,130m,1)] An example of a list constraint is: distances: list of distances with: keep(it == [12km, 13.5km, 70km])

[0099] Resource Types Resource types include junctions and segments and are used to hold the overall list of locations on the current map.

[0100] Complex type MSDL defines three built-in composite types, where scenarios define behaviors such as a car approaching a yield sign or a pedestrian crossing a street. Scenarios define behavior by activating other scenarios. MSDL provides a library of built-in scenarios that describe basic behaviors such as moving, accelerating, and turning. Actors typically represent physical entities in the environment and allow scenarios to define the behavior of physical entities. MSDL provides a set of built-in actors, including car, traffic, and env. If an actor named my_car is instantiated in a program, its built-in scenario, drive, can be invoked as my_car.drive. Structs define a set of related data fields and store values ​​assigned or generated by the program for those fields. For example, a struct might store a car's position, speed, and distance from other cars at a particular time.

[0101] Complex types can be extended to include new data or behaviors, and those definitions can be passed to new complex types through like inheritance. For example, a given actor, car, can be extended to include new scenarios, or a new actor, my_car, can be created that inherits car's scenarios, and then the new scenarios can be added. Complex types can also be conditionally extended using when inheritance. With this function, a type is extended only when a specific condition is true. For example, if the actor my_car has a field of type driving_style, the actor's attributes or behaviors can be different when the value of driving_style is aggressive than when it is normal.

[0102] Prescribed AV type An MSDL environment contains several predefined actors. The actor top contains instances of the following actors: builtin, which represents an MSDL built-in scenario; av_sim_adapter, which represents the MSDL simulator interface; map, which represents a set of paths traveled by actors; traffic, which represents cars, pedestrians, etc.; env, which represents the environmental system and has scenarios such as weather and time of day; and dut, which represents the AV system or device under test. Under traffic, there is a list of car actors called cars. Under dut, a car type dut.car represents the actual dut vehicle (also called ego), and possibly other actors corresponding to various supervisory functions, etc.

[0103] Note that because map, env, traffic, and dut are instantiated as fields in top, these fields can be accessed directly as a global actor, regardless of their location in the hierarchy, for example:

number

number

[0104] The given env actor The env actor is a global actor and contains all environment related activities. For example, the env actor has scenarios that change the environment such as weather and time_of_day, for example:

number

number

[0105] Predefined car actor field The following fields of the car actor can be expanded or constrained to match the allowed types of simulators. These fields can be sampled and used in coverage definitions. [Table 1] The car actor also has a scenario called drive. drive is a driving scenario and has a route parameter that describes the route to be driven. Scenario modifiers can be specified inside the car's drive scenario.

[0106] Predefined enumeration types The following table lists the various enumeration types: [Table 2]

[0107] User Task Flow The verification task flow supported by MSDL is as follows: 1. Plan a validation project. Identify top-level scenario categories that represent risk spectrum, such as urban driving, highway driving, weather, and sensor malfunction. Identify subcategories of scenarios. For example, changing lanes can be a subcategory of highway driving. Identify the behaviors in each scenario subcategory. For example, cutting in and slowing down can be a behavior in the lane change subcategory. Identifying coverage collection points to determine how thoroughly each scenario has been covered (successfully trained). For example, a cut-in and slow-down behavior may have coverage points that include road conditions, distance, and speed. Identify the test criteria (ratings) used to determine how well the DUT performed in various scenarios. · Identify the coverage objectives for those behaviors and scenarios. 2. Generate a verification environment. Describe scenarios, behaviors and coverage points in MSDL based on the lower level built-in scenarios and those available in the library. Identify the DUT and execution platform. · Identify any additional tools that may be used. 3. Automate commissioning. ·Write an exam with a mix of scenarios from different subcategories, such as cutting in and slowing down, possibly with conflicting lane changes. Launch multiple runs with different values ​​for scenario variables such as road conditions, speed and visibility. 4. Analyze the obstacles. Identify the cause of any inspection errors, such as collisions or collision hazards. · Adjust the DUT or apply a temporary patch so that testing can continue. · Automatically rerun all failed runs. 5. Find progress. Analyze the coverage data related to each goal defined in the verification plan to determine which scenarios have not been adequately tested. · Write new exams to cover those uncommon cases.

[0108] The various embodiments disclosed herein may be implemented as hardware, firmware, software, or any combination thereof. Moreover, software is preferably implemented as an application program tangibly embodied on a program storage unit or computer-readable medium comprising part of a particular device or a particular device and / or combination of devices. The application program may be uploaded to and executed by a machine having any suitable architecture. Preferably, the machine is implemented on a computer platform having hardware such as one or more central processing units (“CPUs”), memory, and input / output interfaces. The computer platform may also include an operating system and microinstruction code. The various processes and functions described herein may be either part of the microinstruction code or part of the application program, or any combination thereof, and may be executed by a CPU, whether or not such a computer or processor is explicitly indicated. In addition, various other peripheral units may be connected to the computer platform, such as additional data storage units and printing units. Furthermore, a non-transitory computer-readable medium is any computer-readable medium other than a transitory, propagating signal.

[0109] All examples and conditional language set forth herein are intended for educational purposes to aid the reader in understanding the principles of the disclosed embodiments and concepts contributed by the inventors to further the art, and should not be construed as limiting such specifically recited examples and conditions. Moreover, all statements herein reciting principles, aspects, and examples of the disclosed embodiments, as well as specific examples thereof, are intended to encompass structural and functional equivalents thereof. Additionally, such equivalents are intended to include both currently known equivalents as well as equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure.

[0110] As used herein, the phrase "at least one" in conjunction with a list of items means that any of the listed items may be utilized individually, or any combination of two or more of the listed items may be utilized. For example, a system described as including "at least one of A, B, and C" may include only A, only B, only C, a combination of A and B, a combination of B and C, a combination of A and C, or a combination of A, B, and C.

Claims

1. A method performed by a computer-implemented system for identifying occurrences of at least a scenario, said scenario including at least one object, said method comprising: receiving one or more scenarios, each scenario referencing at least a first object and a second object, each of the first object and the second object being described by the temporal behavior of a first plurality of agents; receiving a data stream of behavior of a plurality of objects, the plurality of objects being described by the temporal behavior of a second plurality of agents; identifying an occurrence of each of the one or more scenarios in the received data stream based on the temporal behavior of the first plurality of agents and the temporal behavior of the second plurality of agents; generating a notification for each identified scenario within the received data stream; A method comprising:

2. The method of claim 1, wherein the object is one of a simulated object and a physical object.

3. The method of claim 1, wherein the first object is a device under test (DUT).

4. The method of claim 3, wherein the DUT is an autonomous vehicle (AV).

5. The step of generating the notification comprises:

2. The method of claim 1, further comprising the step of providing information that enables a reconstruction of a sequence that results in the identification of each identified scenario within the received data stream.

6. The method of claim 1, wherein the object describes at least one of a vehicle, a road, a sidewalk, a person, an animal, a traffic light, a cone, a protective barrier, a bicycle, a train, and a weather element.

7. The method of claim 1, wherein a first scenario of the one or more scenarios includes at least a second scenario of the one or more scenarios.

8. The method of claim 1, wherein the one or more scenarios describe at least one of cutting in front of another object, cutting in from the left lane of another object, cutting in from the right lane of another object, cutting in from two lanes simultaneously, slowing down in front of the object, cutting in front of the object and slowing down, and a change in traffic lights.

9. Generating a notification comprises:

10. The method of claim 1, further comprising generating at least one of a notification upon detection of an event, a key performance indicator (KPI), and coverage information based on identification of each occurrence of the one scenario in the received data stream.

10. The method of claim 1, wherein the data stream includes one or more video segments.

11. The method described in claim 1, wherein the identification of a first occurrence of each occurrence of the one or more scenarios and the identification of a second occurrence of each occurrence of the one or more scenarios are performed in parallel by the computer.

12. The method of claim 1, wherein each scenario of the one or more scenarios is described in measurable scenario description language (MSDL).

13. A non-transitory computer-readable medium storing instructions for causing a processing circuit to perform the method of claim 1.

14. A system for identifying the occurrence of one or more scenarios involving an object, comprising: a network interface; an input / output (I / O) interface; a processing circuit communicatively connected to the network interface, the I / O interface, and a processing unit configured to execute a plurality of instructions provided thereto; a memory containing, in part, instructions for execution, wherein when the processing unit executes the instructions, the system: receiving one or more scenarios, each scenario referencing at least a first object and a second object, each of the first object and the second object being described by the temporal behavior of a first plurality of agents; receiving a data stream of behavior of a plurality of objects, the plurality of objects being described by the temporal behavior of a second plurality of agents; identifying each occurrence of the one or more scenarios in the received data stream based on the temporal behavior of the first plurality of agents and the temporal behavior of the second plurality of agents; generating a notification for each identified scenario within said received data stream; 2. A system configured to:

15. The system of claim 14, wherein the object is one of a simulated object and a physical object.

16. The system of claim 14, wherein the first object is a device under test (DUT).

17. The system of claim 16, wherein the DUT is an autonomous vehicle (AV).

18. The system, 15. The system of claim 14, further configured to generate the notification of further providing information that enables a re-creation of a sequence that results in identification of each identified scenario within the received data stream.

19. The system of claim 14, wherein the physical objects describe at least one of a vehicle, a road, a sidewalk, a person, an animal, a traffic light, a cone, a protective barrier, a bicycle, a train, and a weather element.

20. The system described in claim 14, wherein a first scenario of the one or more scenarios includes at least a second scenario of the one or more scenarios.

21. The system of claim 14, wherein one scenario of the one or more scenarios describes at least one of cutting in front of another object, cutting in from the left lane of another object, cutting in from the right lane of another object, cutting in from two lanes simultaneously, slowing down in front of the object, cutting in front of the object and slowing down, and a change in traffic light.

22. The system, 15. The system of claim 14, further configured to generate at least one of a notification upon detection of an event, a key performance indicator (KPI), and coverage information based on identification of each occurrence of the one scenario in the received data stream.

23. The system of claim 14, wherein the data stream includes one or more video segments.

24. The system described in claim 14, wherein the identification of a first occurrence of each occurrence of the one or more scenarios and the identification of a second occurrence of each occurrence of the one or more scenarios are performed in parallel by the processing circuit.

25. The system of claim 14, wherein each scenario of the one or more scenarios is described in measurable scenario description language (MSDL).

Images