System and method for simulating autonomous vehicle test environment

Abstract

Systems and methods for safely testing a primary autonomous vehicle (AV). The method includes generating a trained machine learning (ML) agent and testing a primary AV in an environment including one or more background vehicles configured to operate in accordance with the trained ML agent. The ML agent is generated by (i) obtaining a test state model having a non-security critical state and a security critical state, (ii) editing the test state model to obtain an edited test state model omitting data about the non-security critical state, and (iii) training an ML agent using the edited state test model to generate the trained ML agent.

Description

Technical Field

[0001] The present disclosure relates to systems and methods for simulating a test environment for an autonomous vehicle. Background Art

[0002] The rapid development of autonomous vehicle (AV) technology is facilitating a new transportation revolution on a scale not seen since the introduction of the automobile a century ago. AV technology has the potential to significantly improve transportation safety, mobility, and sustainability, attracting the attention of industries, government agencies, professional organizations, and academic institutions around the world. Significant progress has been made in the development of AVs over the past two decades, especially with the emergence of deep learning (1). By 2015, many companies had announced that they would mass-produce AVs around 2020 (2–4). So far, reality has not lived up to expectations, with no Level 4 AVs (5) commercially available.

[0003] The reasons for this are multiple. But the most important is that the safety performance of AVs is still lower than that of human drivers, at least for some scenarios. For the average driver in the United States, the probability of an accident in a natural driving environment (NDE) is about 1.9×10 per mile. -6 (6) In comparison, the most advanced AVs have a disengagement rate of approximately 3.3 × 10 per mile, according to California’s 2020 Disengagement Report. -5 (7). Since the disengagement rate can represent safety performance to a certain extent (8,9), this indicates that the gap in safety performance is still more than an order of magnitude.

[0004] A key bottleneck in improving AV safety performance is the severe inefficiency of verifying safety performance. The main method usually tests AV in NDE through a combination of software simulation, (one or more) closed test tracks, and road tests. For example, U.S. Patent Application Publication No. 2020 / 0065443 teaches a simulation system that can be used to test real AVs in a real environment, but enhanced with simulated vehicles. Real traffic information and status information of (one or more) real AVs can be received, for example, at a roadside unit and then sent to a computer system running the simulation. The system therefore coordinates the state and positioning of the real world with those simulated objects (e.g., simulated vehicles) in the simulation. Such a system can be used to test vehicles using simulated elements (e.g., simulated background vehicles).

[0005] However, to validate the safety performance of an AV at the level of a human driver, testing in an NDE would require hundreds of millions of miles, and sometimes hundreds of billions of miles (6). Even with aggressive simulation schemes, it is inefficient; for example, Waymo TMOver the years, only 20 billion miles have been simulated, which is the longest simulation test in the world. Due to this serious inefficiency, AV developers must pay huge economic and time costs to evaluate each new development before they can further improve or deploy it, which seriously hinders the progress of AV development and deployment.

[0006] In essence, verifying the safety performance of AVs is a rare event estimation problem in a high-dimensional space. Figure 1 and Figure 2 As shown in Figure 1, the main challenge is caused by the compound effect of the “curse of rarity” on top of the “curse of dimensionality”. The “curse of dimensionality” refers to the fact that the driving environment can be complex in both space and time, and the variables required to define such an environment have high dimensionality. Since the volume of the variable space grows exponentially with the dimensionality, the computational complexity of many optimization and estimation algorithms may also grow exponentially (10). The “curse of rarity” refers to the fact that the probability of a safety-critical event is very low, that is, most points in the variable space are non-safety-critical and do not provide useful information for such training. In this case, even given a sufficient amount of data, it is difficult for deep learning models to learn because the valuable information of safety-critical events may be buried under a large amount of non-safety-critical data. For example, the policy gradient estimation (11) widely used in deep reinforcement learning (DRL) methods may suffer from too large a variance, thus misleading the training process from the very beginning. It can also lead to severe data imbalance problems with imbalance rates much higher than typical imbalance problems (12). The compound effect of the “curse of rarity” on top of the “curse of dimensionality” (i.e., the rarity of safety-critical events in high-dimensional space) makes the problem even more serious.

[0007] The past few decades have witnessed rapid advances in the problem-solving capabilities of artificial intelligence (AI) systems (13), e.g. Figure 2 As shown, Go’s state space is 10 360 (14), the state space of chip design can be about 10 2500(15). However, prior to this work, simultaneously addressing the “curse of dimensionality” and the “curse of rarity” remains an open problem, which hinders the applicability of AI technology to verify the safety performance of AVs in complex environments. Therefore, all existing methods (counter-proof (16), verification (17,18), and scenario testing (19-25)) can only be applied to spatiotemporally limited scenario segments in the driving environment to evaluate isolated events, such as whether the AV can successfully perform an unprotected left turn. Although these scenario segments are useful for evaluating isolated skills, it may be more critical to evaluate the system safety performance of AVs driving continuously throughout the entire operational design domain, such as safety performance over a full-length trip (26). For example, an AV driving for a long time in a highway environment may interact with many vehicles and other road users (each with different behavioral characteristics) under different weather and lighting conditions, involving various maneuvers such as following, changing lanes, overtaking, merging, etc. Such tests are referred to as “natural safety tests” in this article, which means that such tests are not suitable for evaluating predetermined, isolated safety-critical events. Summary of the invention

[0008] According to one aspect of the present invention, a method for safely testing a host AV is provided. The method includes: generating a trained machine learning (ML) agent and testing the host AV in an environment including one or more background vehicles, the background vehicles being configured to operate according to the trained ML agent. The ML agent is generated by: (i) obtaining a test state model having a non-safety critical state and a safety critical state, (ii) editing the test state model to obtain an edited test state model that omits data about the non-safety critical state, and (iii) training the ML agent using the edited state test model to generate the trained ML agent.

[0009] The method may also include any one of the following features or any technically feasible combination of two or more of the following features:

[0010] - further comprising the step of configuring one or more background vehicles to operate in accordance with the trained ML agent, wherein the configuring step comprises storing data representing the trained ML agent in a computer readable memory;

[0011] - the trained ML agent is a deep reinforcement learning (DRL) agent, and wherein the trained ML agent employs a neural network;

[0012] - the trained ML agent is a dense DRL (D2RL) agent;

[0013] - the test state model is a Markov decision process (MDP), or is based on a Markov decision process (MDP);

[0014] - densify safety-critical data used to train the ML agent using a D2RL approach;

[0015] - the editing sub-step comprises removing said non-safety-critical state and reconnecting the safety-critical state;

[0016] - the environment in which the host AV is tested is a real environment having one or more roads on which the host AV is traveling during the test, and wherein the one or more background vehicles are virtual or simulated vehicles;

[0017] - the testing step comprises performing a simulation synchronized with the host AV and an environment testing the host AV, and wherein the simulation comprises at least one of the one or more background vehicles as a virtual background vehicle; and / or

[0018] -The method is performed by an augmented reality (AR) autonomous vehicle (AV) testing system.

[0019] According to another aspect of the present invention, an autonomous vehicle (AV) testing system is provided. The AV testing system includes: at least one electronic processor and a memory accessible by the at least one electronic processor, wherein the memory stores computer instructions. The AV testing system is configured so that when the at least one electronic processor executes the computer instructions, the AV testing system is used to: generate a trained machine learning (ML) agent and test a host AV in an environment including one or more background vehicles, wherein the background vehicles are configured to operate according to the trained ML agent. The ML agent is generated by the following steps: (i) obtaining a test state model having a non-safety critical state and a safety critical state, (ii) editing the test state model to obtain an edited test state model that omits data about the non-safety critical state, and (iii) training the ML agent using the edited state test model to generate the trained ML agent.

[0020] The AV test system may also include any one of the following features or any technically feasible combination of two or more of the following features:

[0021] - the AV test system is configured such that when the at least one electronic processor executes the computer instructions, the AV test system: configures the one or more background vehicles to operate in accordance with the trained ML agent, wherein the configuring step includes storing data representing the trained ML agent in a computer-readable memory;

[0022] - the trained ML agent is a deep reinforcement learning (DRL) agent, and wherein the trained ML agent employs a neural network;

[0023] - the trained ML agent is a dense DRL (D2RL) agent;

[0024] - the test state model is a Markov decision process (MDP), or is based on a Markov decision process (MDP);

[0025] - densify safety-critical data used to train the ML agent using a D2RL approach;

[0026] - the editing sub-step comprises removing said non-safety-critical state and reconnecting the safety-critical state;

[0027] - the environment in which the host AV is tested is a real environment having one or more roads on which the host AV is driven during the test, and wherein the one or more background vehicles are virtual or simulated vehicles; and / or

[0028] - the testing step comprises performing a simulation synchronized with the host AV and an environment testing the host AV, and wherein the simulation comprises at least one of the one or more background vehicles as a virtual background vehicle.

[0029] According to another aspect of the present invention, a method for safely testing a master AV is provided. The method includes: generating a trained dense deep reinforcement learning (D2RL) agent; configuring one or more background vehicles to operate according to the trained D2RL agent; and after configuring the one or more background vehicles to operate according to the trained D2RL agent, testing the AV in an environment including the one or more background vehicles. The D2RL agent is generated by the following steps: (i) obtaining a test state model having non-safety critical states and safety critical states, (ii) editing the test state model to omit at least one non-safety critical state and reconnect at least two safety critical states, and (iii) training the D2RL agent using the edited state test model to generate the trained D2RL agent. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Illustrative embodiments will be described below with reference to the accompanying drawings, in which:

[0031] Figure 1 It is a schematic diagram of the concepts of the curse of dimensionality and the curse of rarity;

[0032] Figure 2 is a diagram illustrating the compounding effects of the curse of dimensionality and the curse of rarity when they are associated with autonomous vehicles;

[0033] Figure 3depicts an exemplary DRL editing process for obtaining an edited Markov decision process representation, wherein non-critical states are removed and critical states are reconnected, and wherein the edited Markov decision process representation is used to train a neural network to generate a machine learning (ML) agent;

[0034] FIG. 4A to FIG. 4C An exemplary test segment or trip (referred to herein as a “segment”) that can be compiled using the disclosed dense deep reinforcement learning (D2RL) method is shown, where Figure 4A An exemplary segment showing the absence of a critical state is shown, Figure 4B An exemplary segment showing the presence of a critical state without an incident is shown, Figure 4C An exemplary segment showing the presence of critical states separated by non-critical states and an incident;

[0035] Figure 5 An illustration of an augmented reality (AR) testing platform that can augment the real world with virtual background traffic, thereby creating a safer and more controllable and efficient testing environment for AVs, according to at least an embodiment;

[0036] Figure 6 shows the policy gradient estimation of the disclosed D2RL method compared to the DRL method, where the D2RL method removes most of the data of non-informative, non-critical states, which enables unbiased estimation of the policy gradient with significantly reduced variance;

[0037] Figure 7 Bootstrapping is shown, where the disclosed D2RL method is used to effectively propagate information from key states to reduce value approximation error, while the DRL method fails to reduce the error and may even increase the error;

[0038] Figure 8 Depicting a diagram of a corner case generation problem where a neural network controls the maneuvering of eight closest vehicles;

[0039] Fig. 9 Pie charts showing the proportion of data on fragments (left) and steps (right) removed by D2RL are depicted;

[0040] Fig. 10A A graph depicting the amount of training time and reward for a DRL method and a disclosed D2RL method;

[0041] Fig. 10B A graph depicting the amount of training time and accident rate for a DRL method and a disclosed D2RL method;

[0042] Figures 11 to 13Three generated adversarial examples are shown: the first example ( Fig.11 ) demonstrates the ability of the disclosed D2RL method to learn AV responsible accidents. The second example ( Fig.12 ) demonstrates the ability of the disclosed D2RL method to learn the cooperative behavior of multiple vehicles. The third example ( Fig.13 ) demonstrates the ability of the disclosed D2RL method to propagate information between different critical states separated by non-critical states;

[0043] Fig.14A shows a comparison of rewards between the DRL method and the disclosed D2RL method and a graph of the corresponding variance representing the test efficiency of the disclosed D2RL method;

[0044] Fig. 14B shows a graph of accident rates for on-policy and off-policy D2RL methods using the same moving average and standard deviation calculations during training;

[0045] Fig. 14C A graph showing accident rate estimates for a first AV model (AV-I model) obtained using test environments obtained by on-policy and off-policy D2RL methods;

[0046] Fig.14D and Fig.14E The accident rate estimates and relative half-width graphs of the AV-I model are shown respectively;

[0047] Fig.14F A frequency graph showing the number of tests required for a repeated test experiment, wherein the AV-I model is tested by the intelligent test environment;

[0048] Figure 14G and Fig.14H A graph showing the accident rate estimation and relative half-width of the second AV model (AV-II model) obtained by NDE and D2RL-based smart test environment;

[0049] Fig.14I A frequency graph showing the number of tests required for a repeated test experiment, wherein the AV-II model is tested through the intelligent test environment;

[0050] Fig.15 An exemplary master AV that may be tested and may be referred to as a tested AV is shown;

[0051] Fig.16 A diagram illustrating synchronization between a real AV (eg, a host AV) and a virtual background vehicle (BV) is shown;

[0052] Fig.17is a flow chart of a process performed by an image augmented reality (AR) module;

[0053] Fig.18 A diagram showing real-time visualization of the testing process;

[0054] Fig.19A A graph showing accident rate estimates for a real AV at a physical test track with an AR test platform;

[0055] Fig.19B A graph showing the relative half-width of a real AV at a physical test track with an AR test platform;

[0056] Fig. 20 An exemplary simulation platform for use according to one embodiment is shown;

[0057] Fig.21 is a method flow chart for implementing the disclosed D2RL method using an adapted or modified DRL method according to one embodiment;

[0058] Fig. 22 is an illustration of a highway test track at the American Center for Mobility (ACM) that may be used to safely test a host AV, according to one embodiment;

[0059] Fig.23 Describes a system framework that can be used to implement an AV testing system and a method for safely testing a host AV according to one embodiment;

[0060] Fig.24 Depicted is a communication system having an AV test system and a master AV according to one embodiment, wherein the AV test system may be configured to perform the methods disclosed herein; and

[0061] Fig.25 is a flow chart of a method for safely testing a host AV according to one embodiment. DETAILED DESCRIPTION

[0062] The systems and methods described herein are capable of providing simulations for autonomous vehicle (AV) testing of a host AV, which includes introducing safety-critical events to evaluate or test the AV's performance or behavior in response to the introduced safety-critical events. As described above, the "curse of dimensionality" and "curse of rarity" make it particularly difficult to perform accurate, natural testing of safety-critical events for AVs without having to expend a large amount of resources. The disclosed systems and methods can be used to provide simulations with a virtual background vehicle that is directed to operate according to a machine learning (ML) agent (e.g., a dense deep reinforcement learning (D2RL) agent) that causes safety-critical events to be introduced into the test environment as part of natural safety testing of the host AV at a rate that is much higher than when the disclosed methods are not used - that is, the ratio of safety-critical events to non-safety-critical events using the disclosed systems and methods is much higher (i.e., at least 2 times) compared to conventional systems.

[0063] "Safety-critical events" may be defined for specific applications in which the systems and methods are used, but generally and without limitation, a "safety-critical event" for a host AV refers to an event that threatens the safety of one or more passengers of the host AV, and the term "non-safety-critical event" refers to an event that is not a "safety-critical event." Examples of safety-critical events for the host AV are another vehicle cutting in in front of the host AV or another vehicle slamming on the brakes when in front of the host AV. Such maneuvers that cause safety-critical events are referred to herein as adversarial maneuvers. The term "host AV" refers to an AV that is being tested or to be tested using simulation. The term "background vehicle" or "BV" refers to other vehicles, which may be virtual vehicles introduced into the simulation or real vehicles, such as real AVs, that are present at the test facility with the host AV.

[0064] As mentioned above, achieving human-level safety performance for autonomous vehicles remains a challenge. Due to the rarity of safety-critical events in high-dimensional space, i.e., "rare disasters", a major bottleneck is that the cost required to verify the safety of autonomous vehicles in natural driving environments (NDEs) is very high. That is, as part of natural safety testing, it is still difficult to test the safety-critical event processing of AVs, which is similar to a more comprehensive test compared to isolated AV tests designed to test individual test cases. Even given a sufficient amount of data, deep learning models are difficult to learn because the valuable information of safety-critical events is usually buried in a large amount of non-safety-critical data or non-safety-critical events. That is, the ratio of information about safety-critical events to information about non-safety-critical events is very low; however, as a system and method provided herein, the ratio increases dramatically (increased by at least 100 times in at least some embodiments) to overcome the rare disaster and the dimensionality disaster. In some embodiments and implementations, the ratio of safety-critical data to non-safety-critical data (or "safety-critical ratio") can be increased even higher.

[0065] According to at least some embodiments, the systems and methods provided herein address this challenge by implementing a dense deep reinforcement learning (D2RL) method in which a Markov decision process (MDP) is edited by removing non-critical states and reconnecting critical states to each other, so that the use of information in the training data is intensive. Using the disclosed D2RL method, according to at least one embodiment, a background vehicle (BV) is trained in a natural driving environment simulator to learn when to perform specific adversarial maneuvers, thereby generating an intelligent test environment that can maximize evaluation efficiency while ensuring unbiased estimation. According to at least some embodiments, the D2RL method can be extended to spatiotemporally complex driving environments that existing methods cannot handle. The effectiveness of the D2RL method ("disclosed D2RL method") is evaluated by testing a Level 4 AV on a full-length trip in a physical highway test track. According to at least some embodiments and implementations, the disclosed D2RL method accelerates the evaluation process by multiple orders of magnitude compared to testing in NDE.

[0066] In at least some embodiments, the disclosed D2RL method uses dense learning to identify and remove non-safety-critical data, so that only safety-critical data or at least data that is denser in safety-critical data (i.e., has a higher safety-critical ratio) is used to train the neural network. Since only a very small portion of the data is safety-critical, the information of the remaining data will be significantly densified in at least some embodiments or implementations, which overcomes the "rarity curse" and enables the application of deep learning techniques. Specifically, the disclosed D2RL method includes editing a Markov decision process by removing non-critical (or non-safety-critical) states and reconnecting critical (or safety-critical) states to each other, and then using the edited Markov process to train the neural network, such as Figure 3 As shown. Compared with the DRL method, the disclosed D2RL method can significantly reduce the variance of the policy gradient estimate by multiple orders of magnitude without losing unbiasedness, as shown in Theorem 1 below. This significant variance reduction can enable neural networks to learn safety-critical autonomous systems and achieve tasks that are difficult for DRL methods to handle. Using the disclosed D2RL method, background vehicles or other road users are trained by neural networks to learn when to perform what adversarial maneuvers, which is intended to maximize test efficiency and ensure test accuracy. At least according to some embodiments and implementations, the use of this method produces an intelligent test environment that can reduce the test miles required for AV by multiple orders of magnitude compared to the natural driving environment (NDE) while ensuring that the test is unbiased. Therefore, at least according to some embodiments and implementations, the disclosed D2RL method significantly accelerates the evaluation process of AV, which can further shorten the development cycle of AV, and it also opens the door to using advances in artificial intelligence (AI) technology to verify AI performance (i.e., AI testing AI (Figure 4)).

[0067] The effectiveness of the disclosed systems and methods was tested on a 4 km long physical highway test track at the American Center for Mobility (ACM) in Ypsilanti, Michigan, using a real-world Level 4 AV with the open source autonomous driving system Autoware (27). The accident rate of the AV driving continuously on a highway loop was evaluated in the experiment. In order to safely and accurately test the AV using an intelligent test environment, an augmented reality test platform (28) was developed, which combines the physical test track and the microscopic traffic simulator SUMO (29). Figure 5As shown, by synchronizing the movement of the real AV and the virtual background vehicle, the real AV in the physical test track can interact with the virtual background vehicle as if it were in a real traffic environment, where the background vehicle is controlled to interact with the real AV. Since the intelligent test environment is learned based on NDE, the modeled NDE is modeled using large-scale natural driving data from the Safety Pilot Model Deployment Program (30) and the Integrated Vehicle-Based Safety System Program (31) of the University of Michigan, Ann Arbor. Simulation experiments are conducted to fully demonstrate the effectiveness of the disclosed method. Both simulation and field test results show that, at least according to the embodiment, the D2RL method is able to effectively learn the intelligent test environment, which significantly accelerates the evaluation process by multiple orders of magnitude compared to the NDE test method with the same accuracy.

[0068] According to at least one embodiment, an AV test system is provided, the AV test system comprising at least one processor and a memory coupled to the at least one processor. The memory stores computer instructions, which, when executed by at least one processor, cause the AV test system to perform the disclosed method. According to at least some embodiments, a method for safely testing an AV, such as a method for safely testing an AV in a natural environment, is provided, the method comprising: generating a trained machine learning (ML) agent by the following steps: (i) obtaining a test state model having a non-safety critical state and a safety critical state, (ii) editing the test state model to obtain an edited test state model that omits data about the non-safety critical state, and (iii) using the edited state test model to train the ML agent to generate a trained ML agent; and testing a host AV in an environment including one or more background vehicles, the background vehicles being configured to operate according to the trained ML agent.

[0069] According to at least some embodiments, a method of safely testing an AV, such as a method of safely testing an AV in a natural environment, is provided, the method comprising: densifying training data by removing non-safety critical data from the training data; training a neural network using the densified training data to obtain a machine learning (ML) agent; and testing the AV in an environment including one or more background vehicles configured to operate according to the ML agent.

[0070] The following discussion provides an exemplary implementation and discussion of the disclosed D2RL method according to one embodiment. However, certain details and features discussed below will become apparent to those skilled in the art, and the disclosed D2RL method is not necessarily limited to those details and features described below in this exemplary embodiment. The disclosed systems and methods, including the disclosed intensive learning methods performed using the disclosed D2RL method, may be modified or customized according to the desired specific application and / or testing.

[0071] Dense Deep Reinforcement Learning (D2RL). In order to take advantage of advances in artificial intelligence (AI) technology, the test problem can be formulated as a sequential Markov decision process (MDP) (an example of a test state model), where the manipulation of the background vehicle (BV) is decided based on information encoded about the current state of the BV and the host AV or the AV under test (e.g., position and velocity). Based on the MDP formulation, a policy (DRL agent) modeled by a neural network is trained and used to control the manipulation of the BV to interact with the AV, thereby maximizing evaluation efficiency and ensuring unbiasedness. However, as mentioned earlier, due to the "curse of dimensionality" and the "curse of rarity", it is extremely challenging to apply learning-based techniques to safety-critical systems. At least for specific applications, learning effective policies may be difficult or even empirically infeasible if DRL methods are applied directly.

[0072] The disclosed D2RL method addresses these challenges. Due to the rarity of safety-critical events, most states are non-critical and cannot provide information for safety-critical events. According to the disclosed D2RL method, the data of these non-critical states are removed and only informative data is used to train the neural network through policy gradient estimation and bootstrapping. To achieve this, a new definition of non-critical (or non-safety-critical) states is used, such that where q π (s,a) represents the state-action value under the DRL strategy π, represents the state value. It states that if the current action (e.g., manipulation of one or more BVs) does not affect the expected value of the state (e.g., the probability of an AV accident from the state), then the state is defined as non-critical. Note that this definition is mainly used to make the theoretical analysis clear and is not strictly required or necessary to run the algorithm in practice. For example, if the current action does not significantly affect the expected value of the state, then the state can actually be identified as non-critical. For specific applications, critical states can be approximately identified based on specific models or physical properties. For example, in this study, a criticality metric (19, 20) is used to experimentally illustrate the method for the AV testing problem, which is an external approximation of the AV accident rate within a specific time range (e.g., 1 second) from the current state. Note that many other safety metrics (32) may also be applicable, such as the model-predicted instantaneous safety metric developed by the National Highway Traffic Administration (33) and the criticality metric developed by the German PEGASUS project (34), as long as the set of identified states covers critical states.

[0073] After identifying non-critical (or non-safety-critical) states, the Markov decision process is edited to discard or remove the data of the identified non-critical states, and the remaining data is used for policy gradient estimation and bootstrapping for DRL training. FIG. 4A to FIG. 4CThe editing process is shown: the fragments that do not include the key state are completely removed ( Figure 4A ), and the reward from the end state is back-propagated to the key state for policy gradient estimation and bootstrapping ( FIG. 4B to FIG. 4C ). It has been found that dense learning can significantly reduce the variance of policy gradient estimates by multiple orders of magnitude without losing the unbiased nature of the estimate, as shown in Theorem 1 below. Figure 6 The basic concept of the theorem is shown: Since the D2RL method removes most of the data of non-critical states that are not informative, it can estimate the policy gradient unbiasedly with significantly reduced variance. Dense learning can also reduce the bootstrap variance because it can be viewed as state-dependent temporal difference learning (11), where only critical states are utilized and other states are skipped. Figure 7 As shown in Figure 2, since critical states have a much higher probability of causing safety-critical events, the value approximation of critical states usually has a smaller error, which can help reduce the error through bootstrapping. In contrast, non-critical states may even increase the error through bootstrapping due to error accumulation. This mechanism can help avoid interference from a large amount of noisy data and keep the policy learning rare but valuable information.

[0074] To evaluate the effectiveness of intensive learning, the DRL method and the disclosed D2RL method are compared for the extreme case or adversarial example generation problem, which can be formulated as a well-defined reinforcement learning problem. Specifically, a three-lane highway driving environment is studied, where the host AV interacts with dozens of BVs within a certain distance (400m), and each BV has 33 discrete actions every 0.1 seconds: left lane change, 31 discrete longitudinal accelerations ([-4, 2], with a discrete resolution of 0.2ms -2 ) and right lane change. Figure 8 As shown, a neural network is trained to maximize the AV accident rate by controlling the eight closest BVs within 120m. Without using any heuristic-based reward functions or diversification techniques, most existing extreme case generation methods can only be used for scenarios with overly simplified state and action spaces, for example, usually only one BV is controlled as adversarial (16,35). Given a reward for each test segment, i.e., +20 for AV accidents and 0 for other accidents, the parameters of the policy network are updated using proximal policy optimization (PPO) (36). For both the DRL method and the disclosed D2RL method, the neural network is executed for all time steps using the same network structure and hyperparameters. The main difference is that DRL utilizes all data to train the neural network, while the disclosed D2RL method only utilizes data from critical states to train the neural network.

[0075] According to the case study, with DRL ( Fig. 9), implementation of the disclosed D2RL method resulted in the removal of 80.5% of intact fragments not involving critical states and 99.3% of steps from non-critical states. Fig. 10A A comparison of the training rewards between the DRL method and the disclosed D2RL method is shown, where the solid line represents the moving average with a window of 20 and the light shading represents the standard deviation of 5 neighborhood points. Fig. 10B A comparison of the accident rates between the policies learned by the DRL method and the disclosed D2RL method is shown. According to Theorem 1 (below), this shows that, in this case, the disclosed D2RL method can reduce the variance of the policy gradient estimate by about 99.3%, which enables the neural network to go from ineffective learning to effective learning: Fig. 10A As shown in Figure 2, the disclosed D2RL method can effectively learn to increase the reward during the training process, while DRL is stuck from the beginning of the training process; Fig. 10B As shown, the policy learned by the disclosed D2RL method can effectively increase the accident rate of AV, while the policy learned by the DRL method fails to do so. Figures 11 to 13 Three generated adversarial examples are shown: the first example ( Fig.11 ) demonstrates the ability of D2RL to learn AV liability accidents. The second example ( Fig.12 ) demonstrates the ability of D2RL to learn the cooperative behavior of multiple vehicles. The third example ( Fig.13 ) demonstrates the ability of D2RL to propagate information between different critical states separated by non-critical states (see for example Figure 4C ).exist Fig.11 In the example, BV 62 performs an evasive lane change to avoid the forced cut-in of BV 64, but collides with the adjacent BV 66, as shown in accident 68. Fig.12 In the example, BV 72 cuts in, another BV 74 changes lanes to the right, and yet another BV 76 accelerates, which together surround the main AV and cause accident 78. Fig.13 In the example, BV 82 cuts in to force the primary AV to brake, which creates an opportunity for BV 84 to change lanes 2.8 seconds later (i.e., 28 non-critical steps in this embodiment using 0.1 second intervals), which causes accident 88 because BV 86 has to apply the brakes due to the lane change of BV 84. In this case, the cut-in action of BV 82 does not cause any immediate accident, and the only reason for doing so is to create an opportunity for the cut-in action of the following vehicle (BV 84) after 28 non-critical steps.

[0076] Learning Smart Test Environments. A discussion of exemplary implementation aspects of and / or related to the disclosed D2RL methods is provided below. Learning smart test environments for unbiased and efficient AV evaluation may be much more complex than corner case generation. According to importance sampling theory (37), the goal is essentially to learn a new sampling distribution (i.e., importance function) of the manipulations of the BV to replace their natural sampling distribution, which aims to minimize the estimated variance of the AV test. To achieve this goal, a reward function is derived from the estimated variance without using any heuristics or hand-designed functions, as follows:

[0077]

[0078] Where x represents the variable of each test segment, is the indicator function of AV accidents, and is the weight (or likelihood) generated by importance sampling. Here, P(x) represents the natural distribution, and q π (x) represents the importance function of using the target strategy π, Represents the use of behavioral strategy π b For the learning mechanism with the same strategy, π=π b (11). It can be shown that maximizing the reward function in equation (1) is equivalent to minimizing the estimation variance. In order to balance the reward function around zero (which is good practice in reinforcement learning), a positive constant is added, multiplied by a constant, and the function is clipped to obtain the reward function r(x)∈[-100,100], where all these constants can be automatically determined during the learning process. More details can be found below and / or in the attached appendix. Since there is no heuristic or hand-crafted instant reward function, the reward function in equation (1) is highly consistent with the test performance, that is, in this embodiment, a higher reward indicates a more efficient test environment. Since it is mainly based on importance sampling theory, the reward function is also applicable to other rare event estimation problems with high-dimensional variables.

[0079] To determine the learning mechanism, we can study the behavior strategy π b The relationship between π and the target policy π. As proved in Theorem 2 below, the optimal behavior policy is found to collect data during training. It is almost inversely proportional to the target strategy, namely:

[0080]

[0081] where q π* (x) represents the optimal importance sampling function that does not change during training. This shows that if the same strategy learning mechanism (q πb=q π ), the behavior strategy will be far from the optimal, especially at the beginning of the training process (q π Stay away from q π* ), which may mislead the training process and eventually lead to the underestimation problem. To address this issue, an off-policy learning mechanism is designed, in which a general behavior policy is designed and kept unchanged during training. Specifically, the behavior policy controls the manipulation of BVs by following their natural distribution most of the time (with probability 0.99), and controls their manipulations to be adversarial with a small probability (0.01), where the adversarial manipulations can be obtained by the criticality measure. Although this off-policy mechanism is not the optimal behavior policy as in Eq. (2) which is usually unavailable in practice, it can balance exploration and exploitation and is empirically valid for all experimental settings in this study.

[0082] The intelligent test environment is learned by the D2RL method using a reward function and an off-policy learning mechanism. In this study, the PPO algorithm implemented at the RLLib 1.2.0 platform (38) was used. Note that the disclosed D2RL method can be easily applied to most existing DRL platforms and complements most DRL techniques. To provide a training environment, a multi-lane highway driving environment is constructed based on SUMO simulation (29), where all vehicles are controlled every 0.1 seconds using the same action space as in the extreme case generation study. NDE is constructed to provide natural behaviors of BV based on a large-scale natural driving dataset (NDD).

[0083] Empirical Evaluation. An empirical evaluation of an implementation of the disclosed D2RL method according to one embodiment is provided below. To measure safety performance, the accident rate of the AV in the NDE can be used as a benchmark. Since the NDE is generated based on the NDD, it can represent the safety performance of the AV in the real world. In the experiment, the estimated accident rate and the required number of tests of the NDE and the smart test environment can be compared. A single test is performed on the AV at a constant driving distance, and then the test results of the AV (accident or no accident) are recorded, and the accident rate of each test is calculated; the accident rate of each test can be easily converted into an accident rate per mile. To study scalability and versatility, simulation experiments are performed using different numbers of lanes (2 and 3 lanes), constant driving distances (400m, 2km, 4km and 25km) and AV models. Specifically, two different types of AV models are developed: the AV-I model is built based on the intelligent driving model (IDM) (39) and the MOBIL (minimizing overall braking caused by lane change) model (40), while the AV-II model is trained by the DRL technology considering driving efficiency and safety.

[0084] FIG. 14A to FIG. 14I Results are shown for a 2-lane highway driving environment with a driving distance of 400m, which is a basic experiment to validate our approach. FIG. 14A to FIG. 14F The results of the AV-I model are shown. FIG. 14G to FIG. 14I The results of the AV-II model are shown. Fig.14A As shown in Figure 2, during the training process, the estimated variance of the intelligent test environment decreases as the reward function increases, which verifies the consistency between the reward function and the estimated variance in equation (1). In order to prove that the off-policy mechanism is reasonable, the performance of the on-policy mechanism is studied, in which the target policy is used as the behavior policy. Fig. 14B As shown in , the accident rate of the on-policy agent increases significantly during training, while the accident rate of the off-policy agent remains unchanged because the behavior policy does not change. However, this increase in accident rate is misleading because the on-policy mechanism may violate the consistency between the reward function and the estimated variance. Fig. 14C As shown in Figure 2, the test environment obtained by the same policy mechanism underestimates the safety performance of the AV-I model. Differently, the disclosed D2RL method obtains the same accident rate estimate with a much smaller number of tests than the NDE test method, as shown in Figure 2. FIG. 14D to FIG. 14E As shown. To measure the efficiency, the relative half width (19, 20) was calculated as a measure of the evaluation accuracy, and the minimum number of tests to reach a predetermined accuracy threshold (0.3) was calculated. In order to reduce the randomness of the results for fair comparison, the test of the disclosed method was repeated by bootstrap sampling, and the frequency and average of the required number of tests were obtained, as shown in Fig.14F As shown. With the need for 1.9×10 8 Compared with the NDE methods tested, the published D2RL methods only require 1.7×10 5 To investigate the generalizability of the smart test environment, the AV-II model was further tested using the same smart test environment without any improvements, which also yielded accurate and efficient estimates (i.e., approximately one thousand times faster), as shown in Figure 2. FIG. 14G to FIG. 14I shown.

[0085] To further investigate the scalability and versatility of the disclosed D2RL method, experiments were conducted with different numbers of lanes (2 and 3 lanes) and driving distances (400m, 2km, 4km, and 25km). Here, the 25km case was studied because, according to the U.S. Department of Transportation’s Bureau of Transportation Statistics, commuters travel an average of approximately 25km one way, which can demonstrate the effectiveness of the disclosed D2RL method on full-length trips. As shown in Table 1, since the skipped segments and steps significantly reduce the training variance, the disclosed D2RL method can effectively learn the intelligent test environment for all experiments, which can obtain the same accident rate estimate with a much smaller number of tests compared to the NDE test method. Here, the number of tests for the disclosed D2RL method is the average of multiple test experiments, similar to Fig.14F , Fig.14I , and the number of tests of the NDE method is calculated according to the Monte Carlo method (37). It shows the ability of the disclosed D2RL method to overcome the time and space limitations of existing methods, which is of great significance for real-world applications. For example, the highway loop at the American Center for Mobility (ACM) is 2 lanes and 4 km long. The disclosed D2RL method provides the ability to generate an intelligent test environment on this highway environment, which is a task that cannot be accomplished by existing methods.

[0086] FIG. 14A to FIG. 14I The performance evaluation of the disclosed D2RL method is shown. In particular, Fig.14A A comparison of rewards between DRL and D2RL methods is shown, along with a plot of the corresponding variance representing the test efficiency of the D2RL method. The solid line represents the moving average with a window of 20, and the light shading represents the standard deviation of 5 neighborhood points. Fig. 14B Plots showing the accident rates for on-policy and off-policy D2RL methods using the same calculation of moving averages and standard deviations during training. Fig. 14C A graph showing accident rate estimates for AV-I models obtained from test environments obtained by on-policy and off-policy D2RL methods. Light shading indicates a 90% confidence level. Fig.14D and Fig.14E A graph showing the accident rate estimates and relative half-widths for the AV-I model, Figure 14G and Fig.14H The accident rate estimates and relative half-widths of the AV-II model obtained by NDE and D2D-based smart test environments are shown, respectively. The bottom x-axis represents the number of tests for NDE, and the top x-axis represents the number of tests for the smart test environment. Fig.14F and Fig.14I The frequency diagram of the number of tests required for repeated test experiments is shown, where the AV-I model ( Fig.14F ) and AV-II models ( Fig.14I ) are tested through the intelligent testing environment respectively.

[0087] Verification Problem Statement. This section describes the problem statement for safety performance evaluation of an AV according to an implementation or implementation method. The variables of the driving environment are represented as x = [s(0), u(0), u(1), ..., u(T)], where s(k) represents the state (position and velocity) of the AV and the background vehicle (BV) at the kth time step, u(k) represents the manipulation of the BV at the kth time step, and T represents the total time step of the test segment. Using the Markov Assumption of the BV's manipulation, the probability of each test segment in the natural driving environment can be calculated as The AV accident rate can then be measured by the Monte Carlo method (32) as:

[0088]

[0089] Where A represents the accident event, n represents the total number of test clips, and x i ~P(x) indicates that the variable is sampled from the distribution P(x). Since A is a rare event, obtaining a statistically reliable estimate requires a large number of tests (n), which leads to serious inefficiencies in the on-road testing approach, as pointed out in (6).

[0090] To address this inefficiency, an intelligent driving environment is generated in which the BV can be purposefully controlled to test the AV unbiasedly and efficiently. Essentially, testing the AV in the intelligent driving environment is done by estimating P(A) in equation (3) via the importance sampling method (37), as follows:

[0091]

[0092] where q(x) represents the underlying distribution of the BV manipulation in the smart testing environment, W q (x) is the probability of each test segment, as follows:

[0093]

[0094] According to the importance sampling theory (32), if q(x)>0 for any x where P(A|x)P(x)>0, then the unbiasedness of the estimate in equation (4) is guaranteed. To optimize the estimation efficiency, the importance function q(x) needs to minimize the estimation variance

[0095]

[0096] Therefore, the generation of the intelligent test environment is formulated as a sequential Markov decision process (MDP) problem of the manipulation of the BV (i.e., determining q(u(k)|s(k))) such that the estimated variance in Eq. (6) Minimize. However, how to solve this sequential MDP problem associated with rare events and high-dimensional variables remains a very challenging problem, and most existing importance sampling-based methods suffer from the "curse of dimensionality" (43), in which the estimated variance will increase exponentially with the variable dimension. In previous studies (22), it was found that the "curse of dimensionality" problem can theoretically be solved by rare adversarial control of natural distributions. However, only model-based methods using handcrafted heuristics have been used for rare adversarial control, which suffers from obvious time and space limitations, and traditional methods cannot use AI technology to train BV to truly learn test intelligence.

[0097] Formulation of Deep Reinforcement Learning Problem. This section describes how to generate a smart test environment as a DRL problem, at least according to one embodiment. As described above, at least in some embodiments, the goal is to minimize the estimated variance in Eq. (6) by training a policy π modeled by a neural network θ that can exploit the underlying distribution q π (u|s) to control the manipulation of the BV. To keep the notation simple, it is implicit in all cases that π is a function of θ. MDPs are usually composed of four key elements: state, action, state transition, and reward. In this study, the state encodes information (position and velocity) about the AV and the surrounding BVs (the closest eight vehicles within 120m), and the action includes 31 discrete longitudinal accelerations ([-4,2], with a discrete resolution of 0.2ms -2 ), left lane change, and right lane change, the state transitions define a probability distribution over the next state that also depends on the AV's maneuver. Of course, the learning problem and method can be applied to other BV actions or maneuvers and / or actions or maneuvers of other road users. To keep the runtime of the DRL small, the output of the neural network is simplified to the adversarial maneuver probability (ε) of the most critical BV (i.e., the dominant other vehicle or "POV"). π ), while other manipulations of POV follow the natural distribution through 1-ε π is normalized, and the manipulations of the other BVs are kept to follow the natural distribution. The adversarial manipulations and POVs are determined by the criticality metric. Note that generalizing this work to multiple POVs is straightforward, leveraging more computational resources. To demonstrate the generality of the approach, the action space of the manipulations of eight POVs is kept in the extreme case generation problem.

[0098] Reward function design is crucial to the DRL problem (44). Since the goal of the intelligent test environment is to minimize the estimation variance in Equation (6), the objective function of the DRL problem is derived as:

[0099]

[0100] in, is the indicator function of the accident event, π b represents the behavior strategy of DRL, π b =π is used for the same-policy learning mechanism. During the training process, the training data is collected through the behavior strategy, which is the Monte Carlo estimate of the expectation in equation (7), so the following reward function can be obtained:

[0101]

[0102] In theory, it is consistent with the objective function. In order to make the reward function balanced near zero (which is a good practice for reinforcement learning), a positive constant is added, multiplied by a constant, and the function is clipped to obtain the reward function r′(x)∈[-100,100], that is, r ′(x) = clip[C1·r(x)+C2,-100,100], where all these constants C1 and C2 can be automatically determined during the learning process to balance the rewards. Note that this approach does not use any heuristics or hand-crafted functions for reward function design, which can be misleading.

[0103] Dense Deep Reinforcement Learning (D2RL). With states, actions, state transitions, and reward functions, the intelligent test generation problem becomes a DRL problem, which has the challenges of "curse of dimensionality" and "curse of rarity". To address this challenge, the disclosed D2RL method can be used. Specifically, according to the policy gradient theorem (11), the policy gradient of the objective function of the DRL method can be estimated as:

[0104]

[0105] Where θ represents the parameters of the strategy, q π (S t ,A t ) represents the state-action value, S t and A t is a sample of states and actions under the policy, Yes π (S t ,A t ), that is, Differently, for the D2RL method, it is proposed to estimate the policy gradient as:

[0106]

[0107] in represents a collection of key states, represents the indicator function. Here, if This state is defined as a non-critical state, where represents the state value, so the key state set can be defined as It states that a state is defined as non-critical if the current action (e.g., manipulation of the BV) will not affect the expected value of the state (e.g., AV accident probability from the state).

[0108] Theorem 1: The policy gradient estimator of D2RL has the following properties:

[0109] (1)

[0110] (2) as well as

[0111] (3) Assumptions

[0112]

[0113] in is the proportion of critical states among all states under policy π (see for example Fig. 9 ),

[0114] Theorem 1 shows that the proposed D2RL method is unbiased and efficient in estimating the policy gradient compared to the DRL method. To quantify the variance reduction of dense learning, we introduce the assumption in Eq. (11), which assumes Independent of indicator function Since the policy and state-action values ​​are initialized randomly, for all different states, the value are very similar, so this assumption is valid in the early stages of the training process. This significant variance reduction will enable the disclosed D2RL method to optimize the neural network, while the DRL method will be stuck at the beginning of the training process. Note that the definition and assumption of non-critical states are mainly for the purpose of making the theoretical analysis clear and may not be strictly required to run the algorithm in practice. For example, if the current action does not significantly affect the expected value of the state, the state can actually be identified as non-critical. Those skilled in the art will understand that identifying non-critical (or non-critical) states or critical states or defining what constitutes non-critical (or non-critical) states or critical states is defined according to the application or implementation in which the disclosed D2RL method is to be used, because the specific boundaries of what constitutes a critical state can be selected based on the specific application in which the disclosed D2RL method is used.

[0115] Dense learning for estimation using bootstrapping The influence of can guide the information propagation in the state-action space. For example, the fixed-length advantage estimator The following is usually used in the PPO algorithm (36):

[0116]

[0117] where δ t =r t +γV(s t+1 )-V(s t ), V(s t ) is the state value function, and L represents a fixed length. For safety-critical applications, the immediate reward is usually zero (i.e., r t =0), and due to the rarity of the event, most of the state value functions are determined by initial random values ​​without any valuable information. Using such a noisy state value function will not guide the learning process, but will hinder valuable information. By editing the Markov chain, only the critical states will be considered. Then, the advantage estimator will be basically modified as:

[0118]

[0119] Among them, δ z(t,j) =r z(t,j) +γV(s z(t,j+1) )-V(s z(t,j) ), z(t,0)=t, Essentially, it is state-dependent temporal difference (TD) learning, where only the values ​​of critical states are used for bootstrapping. Since critical states have a much higher probability of leading to safety-critical events, reward information can be more easily propagated to these critical state values. With the values ​​of these critical states, bootstrapping can more effectively guide information from safety-critical events to the state-action space.

[0120] Due to the above-mentioned variance reduction on policy gradient estimation and bootstrapping, the disclosed D2RL method significantly improves learning effectiveness compared to the DRL method, enabling neural networks to learn from safety-critical events. It opens the door to using AV technology for safety-critical autonomous systems, including the corner case generation problem and the smart test environment generation problem in this study.

[0121] At least in some embodiments, the behavior strategy π b The goal is to collect training data to improve the target policy π, which can maximize the objective function in equation (7). To achieve this goal, it is crucial to accurately estimate the objective function using the reward function in equation (8), which determines the calculation of the policy gradient. However, due to the rarity of accidents, only rare fragments have non-zero rewards, so the objective function estimate has a large variance. Without an accurate estimate of the objective function, training may be misled. According to the importance sampling theory, the following theorem is provided:

[0122] Theorem 2: At least in some embodiments, the optimal behavior policy that minimizes the estimated variance of the objective function is It has the following properties:

[0123]

[0124] where q π* (x) represents the optimal importance sampling function that does not change during training, and the symbol ∝ means “proportional to…”.

[0125] Theorem 2 finds that the optimal action strategy is almost inversely proportional to the target strategy, especially in q π Stay away from q π* When the training process begins. When the optimal target strategy has been obtained, that is, q π =q π* According to Theorem 2, we can get Theorem 2 shows that if the learning mechanism with the same strategy is used Then the behavior policy will be far from optimal, which may mislead the training process and eventually lead to underestimation problems. For example, if the target policy omits actions that may lead to possible accidents, the same-policy learning mechanism will never discover such omitted accidents. More importantly, the same-policy mechanism may mislead the policy used to deliberately hide accidents that are difficult to evaluate, leading to a serious underestimation problem in safety performance evaluation.

[0126] This problem can be addressed using an off-policy learning mechanism, in which a general behavior policy is designed that remains constant during training. Specifically, a constant probability of adversarial manipulation of POV is determined (i.e., ), and the other manipulations performed with total probability 0.99 are normalized according to the natural distribution. The policy uses the natural distribution to explore the state-action space most of the time, and leverages information from the model-based criticality measure to help identify POV and adversarial manipulations. Note that although it may be necessary to adaptively determine the optimal behavior policy based on the target policy, as indicated in Theorem 2, the off-policy learning mechanism can provide a good enough basis for effective learning in this study. The behavior policy is constant It is also not sensitive, and typically large values ​​that balance exploration and exploitation (e.g., 0.9, 0.95, 0.99, etc.) will be effective in this study.

[0127] Table 1

[0128]

[0129] According to one embodiment, the effectiveness of the disclosed D2RL method was demonstrated by testing a hybrid Lincoln on a physical multi-lane 4 km highway test track at ACM. TM MKZ TM It has been proven that the hybrid Lincoln TM MKZ TM Equipped with open source autonomous driving system Autoware(27) and Fig.15 The goal is to continuously drive the AV on a highway loop and evaluate its accident rate. Fig.15 In the , “RTK” stands for real-time kinematic positioning; “IMU” stands for inertial measurement unit; “DSRC” stands for dedicated short-range communication; and “OBU” stands for on-board unit. In order to test AVs using a smart test environment, an augmented reality test platform (28) was developed that combines a physical test track and a simulated environment SUMO (29).

[0130] like Fig.16As shown, by synchronizing the movement of the real AV and the virtual BV, the real AV in the physical test track can interact with the virtual BV as if it were in a real traffic environment, where the BV acts with the real AV according to the intelligent test environment. An AV test system that can be configured to use the disclosed D2RL method to determine the actions (e.g., maneuvers) of one or more BVs can receive traffic state and associated vehicle state information from the environment in which the AV is being tested. For example, Fig.16 As shown, traffic and AV states can be input into a neural network to determine intelligent traffic behavior, which can refer to the actions of the BV determined using the neural network. As discussed herein, the disclosed D2RL method can be implemented using DRL technology, including using a DRL agent trained using a D2RL method (e.g., the disclosed D2RL method), which is referred to herein as a trained D2RL agent. Fig.16 Also shown is virtual traffic information being sent to real vehicles. In at least some embodiments, the virtual traffic information includes information related to one or more BVs and may include one or more virtual or simulated basic safety messages (sBSMs), such as the sBSMs described in paragraphs

[0011] ,

[0019] ,

[0033] ,

[0062] , and

[0081] -

[0086] of U.S. Patent Application Publication No. 2020 / 0065443, which is incorporated herein by reference.

[0131] The system was implemented at the American Center for Mobility (ACM) with an average communication latency of 32.60ms, which is acceptable for AV testing. To implement the test camera-based perception and positioning system, an image augmented reality module was developed that can overlay a virtual BV onto the real camera video in real time, such as Fig.17 shown. Fig.18 A real-time visualization of the testing process is shown, and more details can be found in Supplementary Movies 2 and 3. Fig.18 , the leftmost figure shows a simulated view in which a virtual BV 90 is generated and controlled by the smart test environment to interact with a master AV 92; the middle figure shows a real-world AV view visualized by Autoware, together with the master AV 92 (or AV under test) and the virtual or augmented BV 90; the rightmost image shows an original image view (top) and an augmented image view (bottom) from a front camera of the master AV 92.

[0132] The smart test environment in the digital twin of the ACM highway is trained using the same training setup as the simulation study. However, as shown in Table 1, while the smart test environment can reduce the AV testing time from about 10 8 The test loop was accelerated to only about 4.4×10 4In order to demonstrate the disclosed D2RL method in a more efficient way, the NDE is simplified without loss of generality, the smart test environment is trained accordingly, and the main AV is tested. Fig.19A A graph showing accident rate estimates for a real AV at a physical test track with an augmented reality test platform. Fig.19B A graph showing the relative half-width of a real AV at a physical test track with an augmented reality test platform. The black dashed line ( Fig.19A ) represents the final estimate of the accident rate, and the light dashed line ( Fig.19A ) indicates a relative half-width threshold of 0.3, and light shading indicates a 90% confidence level. Fig.17 and Fig.19B As shown in Figure 2, the estimated value of the accident rate converges after about 161 tests and reaches a relative half-width of 30%, which is significantly smaller than the estimate of the NDE method (2.5×10 7 ).

[0133] The results provide evidence for the use of DRL technology to verify the safety performance of AVs, which opens the door to using AI technology to verify the machine intelligence of safety-critical autonomous systems. The intensive learning method can enable AI technology to be used for a wide range of safety-critical applications with high-dimensional variables, while overcoming the "curse of dimensionality" and "curse of rarity". It can significantly enhance existing testing methods (e.g., disproof, formal methods, and scenario testing) to overcome their limitations in real-world applications. In the study, the effectiveness of the intensive learning or D2RL method for disproof was demonstrated, which involved scenario testing of multiple adversarial maneuvers of the vehicle and complex trip-level driving environments at high temporal resolution, tasks that existing methods cannot handle. Modeling of intelligent test environment generation provides a basis for future AI-based machine intelligence verification research. Intelligent test environments can be used to enhance existing life-like simulations to accelerate the testing and training process of AVs. Extreme cases can also be used for further development of AVs. Using realistic NDE models, the disclosed D2RL method can convert each test mile in the test track into an equivalent of hundreds or even thousands of driving miles on public roads, which can significantly accelerate the testing process of AVs. The entire framework can also be applied to machine intelligence verification of other safety-critical autonomous systems with similar characteristics.

[0134] Densifying information is used to overcome the challenges posed by rarity. In the field of deep neural networks, it has been demonstrated that more densely connecting different layers of a neural network has better training efficiency and efficacy (i.e., DenseNet (41)). In addition to the natural connections provided by state transitions, the disclosed D2RL method densifies information by connecting states more densely to safety-critical states, rather than connecting layers of a neural network. As safety-critical states are more associated with rare events, they have more valuable information. By densifying the connections between safety-critical states and other states, valuable information can be better propagated throughout the state space, which can significantly facilitate the learning process. According to one embodiment, non-critical states can be roughly identified and the remaining states directly connected. According to some embodiments, this can be further improved by more flexible and dense connections, such as between safety-critical states and non-critical states, because the identified non-critical states may not be strictly non-critical. Connections can even be added in the form of curriculum learning (42), which can gradually guide information propagation. Measures for identifying critical states can also be further improved by involving more advanced modeling techniques.

[0135] An augmented reality testing platform may be used because, at least according to some embodiments, it provides a promising way to test physical AVs safely, accurately, and efficiently. The framework, including the disclosed D2RL method, may also be applied to testing general safety-critical autonomous systems. Since the credibility of test results depends on the fidelity of the NDE model in the digital twin, it may be critical to build NDE models with statistical and interactive realism, which can be further improved by leveraging high-resolution, large-scale natural driving data. To further enable testing of LiDAR and radar-based capabilities, we will also develop capabilities to augment data from these sensors. The platform also provides a promising environment for training AVs, which will be further studied. This may create a symbiotic relationship between testing and training of safety-critical autonomous systems, accelerating both areas.

[0136] Simulation Setup. Following is a discussion of a natural driving environment (NDE) simulator that may be used as part of the AV testing systems discussed herein, according to at least one embodiment.

[0137] Natural Driving Environment Simulator. To build digital twins for training and testing, a simulation platform was developed based on the open source traffic simulator SUMO and the open source vehicle simulator CARLA, in which the advantages of these simulators are combined, which significantly expands their compatibility with high-fidelity driving environments (including natural driving environments and smart test environments). Fig. 20An exemplary simulation platform used is shown in . SUMO can provide a convenient way for scalable network construction, multimodal traffic, traffic demand modeling and traffic management, while CARLA can implement high-fidelity vehicle dynamics, multiple sensor models, high-definition maps, and integrated interfaces with high-fidelity autonomous driving systems (such as Autoware). Both C++ and TRACI interfaces are used to improve the SUMO simulator so that a high-fidelity driving environment can be integrated and CARLA can be synchronized. Specifically, the C++ code of SUMO (including vehicle following and lane change behavior models) for integrating a high-fidelity driving environment is rewritten and recompiled. Then, the TRACI interface is used to implement an intelligent test environment, where at a selected moment, a selected vehicle will execute a specific adversarial mover according to the strategy modeled by the neural network. The modified SUMO and CARLA simulators synchronize information about background vehicles, autonomous driving vehicles, traffic signals, high-definition maps, etc. through the TRACI interface.

[0138] Natural Driving Environment Model. In this study, a Natural Driving Environment (NDE) model was constructed based on the large-scale Natural Driving Dataset (NDD) from the Safety Pilot Model Deployment Program (30) and the Integrated Vehicle-Based Safety Systems Program (31) of the University of Michigan, Ann Arbor to provide the natural behavior of the BV. To build the NDE model, approximately 2.95×10 8 data points, which are approximately 8,200 driving hours. Given the state-encoded information of the current traffic, the NDE model can provide a distribution of maneuvers for each BV that is consistent with the NDD. Then, by sampling maneuvers from the distribution, a test environment that can represent real-world safety performance can be generated. More details of the NDE modeling approach can be found in (45). For field testing at the ACM, the NDE model was simplified to demonstrate our approach more efficiently, as simpler NDEs typically require fewer tests in an intelligent test environment. Specifically, the Intelligent Driving Model (IDM) (39) and the MOBIL (Minimizing Overall Braking Induced by Lane Change) model (40) were modeled as stochastic models, which were used as simplified NDE models.

[0139] D2RL architecture, implementation, and training. By defining a specific environment using dense learning methods, the D2RL algorithm can be easily plugged into the current DRL algorithm. Specifically, for the current DRL algorithm, the environment receives decisions from the DRL agent, executes the decision, and then collects observations and rewards at each time step, while for the D2RL algorithm, the environment only collects observations and rewards for key states, such as Fig.21As shown. In this way, the D2RL method can be implemented using existing DRL platforms. In this study, the PPO algorithm implemented in the RLLib 1.2.0 platform (38) was used, which was trained in parallel on a high-performance computing cluster with 500 CPU cores and 3500GB of memory at the University of Michigan, Ann Arbor. A three-layer fully connected neural network with 256 neurons in each layer was designed. In addition to the default parameters, 10 -4 The learning rate was 0.000 and the discount factor was 1.0. Each CPU collected 120 time steps of training data for all experimental settings in each training iteration, so a total of 60,000 time steps were collected in each training iteration. To provide a training environment, a multi-lane highway driving environment was built on a natural driving environment simulator, where each vehicle was controlled every 0.1 seconds with 33 discrete action spaces: left lane change, 31 discrete longitudinal accelerations ([-4, 2], with a discrete resolution of 0.2ms -2 ) and right lane change. For extreme case generation, the output of the neural network is the actions of the eight closest BVs, each of which has the 33 discrete action spaces mentioned above. For intelligent test environment generation, the output of the neural network is the adversarial manipulation probability (ε π ), where the action space is discretized into 0.1, 0.2, …, 0.9, 0.91, 0.92, …, 0.99, 0.995, 0.999. The generalization to continuous action space is straightforward.

[0140] Field Test Setup. An exemplary field test setup and related discussion according to one embodiment and implementation is provided below.

[0141] The augmented reality testbed was implemented at the American Center for Mobility (ACM), a closed CAV testing facility in Ypsilanti, Michigan. In the study, a 4km highway loop was utilized, which had two-three lanes and exit and entrance ramps to create various merging opportunities, such as Fig. 22As shown. A digital twin of the ACM based on a natural driving simulator and a high-definition map was built. In order to synchronize the information between the simulation and the physical test track, 12 Siemens dedicated short-range communication (DSRC) roadside units (RSUs) installed along the highway loop were utilized. These DSRC devices can communicate with AVs on the highway via the 802.11p and SAE J2735 protocols through the immediate forwarding message (IMF) and forwarding functions. Specifically, the IMF function is used to broadcast a proxy basic safety message (BSM) containing the identifier, latitude, longitude, altitude, etc. of the virtual background vehicle to the physical AV, while the forwarding function is used to forward the BSM of the incoming AV to the digital twin. After receiving the BSM of the AV, the AV state in the simulated world is synchronized, where the background vehicle is controlled by the intelligent test environment.

[0142] Augmented Image Rendering. Augmented reality is used to render and blend virtual objects (e.g., vehicles) onto the camera view of the ego vehicle. Given a background 3D model with its 6DoF pose / position in world coordinates, a two-stage transformation is performed to project the model onto the onboard camera ( Fig.17 ): 1) from world coordinates to ego vehicle coordinates, and 2) from ego vehicle coordinates to on-board camera coordinates. In the first transformation, the pose and position of the ego vehicle are obtained from the real-time signal of the on-board high-precision RTK. In the second transformation, the projection is based on the pre-calibrated camera intrinsics and extrinsics. Relighting is performed on the rendering layer to coordinate the visual quality of the blending result. An enhanced view is generated based on linear blending with the rendered foreground layer, the camera's background layer, and the rendered alpha matte. On top of the blending result, a weather control layer is further added to simulate different weather conditions, such as rain, snow, and fog. Enhanced rendering is implemented based on pyrender(46). The 3D model assets are selected from the ShapeNet dataset.

[0143] Self-driving vehicles. Mcity from the University of Michigan, Ann Arbor, as the host AV or the AV under test TM Test facility for the modified Lincoln TM MKZ TM Equipped with multiple sensors, computing resources (2Nexcom TM Lumina TM ) and by Dataspeed TM Co., Ltd. provides drive-by-wire capabilities. Specifically, the sensors include PointGrey TM Camera, Velodyne TM 32-channel LiDAR, Delphi TMRadar, OTXS RT3003 RTK GPS, Xsens MTi TM GPS / IMU, etc. The main AV is implemented using the ROS-based open source software Autoware AI (27), which provides a full-stack software for highly automated driving functions including positioning, perception, planning, control, etc. The main AV is then integrated with the AR test platform to evaluate the safety performance of the AV. Fig.23 A diagram of the system framework according to one embodiment is shown in FIG. Specifically, the AV localization component is modified to utilize high-definition maps and high-precision RTK to obtain the current pose and velocity. The BSMs of surrounding vehicles are directly obtained from the simulation via wireless communication. To generate the future trajectory of the AV, OpenPlanner TM 1.13(47) is used as the decision module, which is an advanced planning algorithm that includes global and local path planning. A pure pursuit algorithm is applied to convert the planned trajectory into velocity and yaw rate, which are then converted into the velocity and yaw rate using the Dataspeed TM The PID controller provided by the company further converts them into vehicle drive-by-wire control commands, namely steering angle, throttle and braking percentage.

[0144] Now refer to Fig.24 , an operating environment is shown, which includes a communication system 100, a host autonomous vehicle (AV) 102 with vehicle electronics 104, a real background vehicle (BV) 106, (one or more) roadside units 108, a backend server 112, a land network 114, a wireless carrier system 116, and a global navigation satellite system (GNSS) satellite constellation 118. The communication system 100 includes an AV test system 110, which is composed of at least one processor and a memory coupled to the at least one processor, as discussed in more detail below. In some embodiments, the real BV 106 can be omitted, such as in the case where all BVs used for testing are virtual BVs. As used herein, a real BV is a BV that is a real physical vehicle and a virtual BV is a BV simulated using a computer simulation. It should be understood that although Fig.24 The illustrated embodiment provides one example of one such communication system 100 , but the systems and methods described below may be used as part of a variety of other communication systems.

[0145] Land network 114 may be a conventional land-based telecommunications network that is connected to one or more landline telephones and connects wireless carrier system 116 to backend server 112. For example, land network 114 may include a public switched telephone network (PSTN), such as that used to provide hardwired telephone, packet-switched data communications, and Internet infrastructure. One or more segments of land network 114 may be implemented using standard wired networks, fiber optic or other optical networks, cable networks, power lines, other wireless networks (e.g., wireless local area networks (WLANs)), or networks that provide broadband wireless access (BWA), or any combination thereof.

[0146] The wireless carrier system 116 may be any suitable remote data transmission system, such as a cellular telephone system. The wireless carrier system 116 is shown as including a single cellular tower 120; however, the wireless carrier system 116 may include additional cellular towers and one or more of the following components, which may depend on the cellular technology used: base transceiver stations, mobile switching centers, base station controllers, evolved nodes (e.g., eNodeBs), mobile management entities (MMEs), serving and PGN gateways, etc., as well as any other networking components for connecting the wireless carrier system 116 with the land network 114 or connecting the wireless carrier system 116 with user equipment (UE, for example, which may include a telematics device in the main AV 102 or the real BV 106), all of which are generally represented by 122. The wireless carrier system 116 may implement any suitable communication technology, including, for example, GSM / GPRS technology, CDMA or CDMA2000 technology, LTE technology, 5G, etc. In at least one embodiment, the wireless carrier system 116 implements 5G cellular communication technology and includes suitable hardware and configuration. In some such embodiments, the wireless carrier system 116 provides a 5G network that can be used by the host AV 102 and / or the real BV 106 to communicate with a backend server 112 or other computer / device located remotely from the host AV 102 and / or the real BV 106. In general, the wireless carrier system 116, its components, the arrangement of its components, the interaction between such components, etc. are well known in the art.

[0147] The backend server 112 may be used to provide a backend for one or more components of the host AV 102 and / or the real BV 106, for example, to receive data from the vehicles 102, 106 for executing the following method 200 ( Fig.25) and / or for other purposes, such as for sending over-the-air (OTA) updates to the vehicles 102, 106. In at least one embodiment, the backend server 112 includes one or more computers or computing devices (collectively referred to as "computers"). In some embodiments, the backend server 112 is used to store information related to the vehicles 102, 106, such as vehicle status information that can be used to evaluate the performance of the master AV 102. The backend server 112 is a server executed or hosted by one or more computers, each of which includes a processor and a non-transitory computer-readable memory accessible by the processor. In at least one embodiment, the backend server 112 is used to provide configuration information (e.g., data, computer instructions) used as part of the AV testing of the master AV 102. The configuration information can be used to guide the actions of one or more BVs present during the testing of the master AV 102. In some embodiments, the configuration information causes the BV to operate according to a trained machine learning (ML) agent, which can be a trained D2RL agent as described below. The configuration information can be used by a simulator that simulates the BV and / or its actions / movements. In one embodiment, an ML agent is trained using the disclosed D2RL method and then used to determine the actions of one or more BVs present during testing of the master AV 102. The training of the ML agent, including the compilation of a Markov decision process (MDP) discussed in more detail below, may be performed by the backend server 112.

[0148] It should be understood that the backend server 112 may actually include multiple servers that may be collocated or remotely located from each other and may be used for a variety of services. In some embodiments, one or more steps or functions described herein as being performed by the backend server 112 may be performed by a cloud computing service, and the actual processing may be performed in a distributed manner. However, it should be understood that the backend server 112 remote from the main AV 102 may be implemented in a variety of ways and is not limited to any particular configuration, type, or number of computing devices.

[0149] The backend server 112 is shown to include one or more processors 150 and non-transitory computer-readable memory 152. In some embodiments, the backend server 112 can be configured so that when the computer instructions stored on the memory 152 are executed by the processor(s) 150, the backend server 112 causes certain steps and / or functions to be performed, such as any of the functions discussed herein as being attributed to the backend server 112. The backend server 112 can be used for various purposes, such as providing vehicle information about the road it is operating on and / or other information about the vehicle environment. In some embodiments, the communication system 100 includes various backend servers for various purposes.

[0150] In at least one embodiment, the backend server 112 is used to perform the method 200 described below: Fig.25 ). For example, in one embodiment, the memory 152 stores computer instructions that, when executed by at least one processor 150, cause the method 200 ( Fig.25 ) is performed. And, as described above, the back-end server 112 can be composed of various computing devices that can be co-located with each other or located remotely from each other. In one embodiment, a first computer subsystem of the back-end server 112 can be used to perform the Markov decision process editing steps herein, and another computer subsystem of the back-end server 112 can be used to train a neural network using the edited Markov decision process. Then, for example, a third computer subsystem can be used to execute an ML agent to control the operation of one or more BVs (e.g., a real BV 106 and (one or more) virtual BVs). However, in other embodiments, such steps or functions may be performed by the same computing device or computer subsystem or by other components of the communication system 100.

[0151] The host AV 102 is depicted as a passenger vehicle in the illustrated embodiment, but it should be understood that any other vehicle may be used, including motorcycles, trucks, sport utility vehicles (SUVs), recreational vehicles (RVs), other vehicles or mobile devices that can be used on roads or sidewalks, etc. As depicted in the illustrated embodiment, the host AV 102 includes vehicle electronics 104. The vehicle electronics 104 includes an onboard vehicle computer 130, a GNSS receiver 132, one or more sensors 134, and a vehicle communication bus 136. Fig.24 Examples of certain components of the vehicle electronics 104 are provided; however, it should be understood that according to various embodiments, the vehicle electronics 104 may include one or more other components as Fig.24 in addition to or as a substitute for those components depicted in .

[0152] A real background vehicle (BV) 106 is a real vehicle, similar to the main AV 12, and may include hardware that is the same or similar to the hardware of the main AV 12. Each component of the main AV 102 is thus attributed to the real BV 106. The real BV 106 may include different configurations or operating instructions than the main AV 102, because the main AV is the AV under test, and the real BV 106 is a BV used to test the main AV. The real BV 106 may be at least partially controlled by the methods discussed herein, which may be based on the disclosed D2RL methods. In some embodiments, the AV test system 110 includes using the real BV 106 to test the main AV 102. As described above, at least in some embodiments, the real BV 106 is omitted.

[0153] Referring back to the vehicle electronics 104 of the master AV 102, a global navigation satellite system (GNSS) receiver 132 receives radio signals from the GNSS satellite constellation 118. The GNSS receiver 132 uses the received radio signals to generate position data that indicates the location of the GNSS receiver 132, and therefore the location of the master AV 102 on which it is installed. In one embodiment, such as where the system 100 is located in the United States, the GNSS receiver 132 may be a global positioning system (GPS) receiver. In another embodiment, such as where the system 100 is located in Europe, the GNSS receiver 132 may be a GNSS receiver configured for use with Galileo. In addition to the position data that may indicate the location as a pair of geographic coordinates, the GNSS receiver 132 may also specify a time associated with each location. Such time and position data obtained by the GNSS receiver based on the GNSS signals is referred to as GNSS data. The position data may be used in conjunction with other data, such as map data that specifies road attributes, to identify information about the road on which the vehicle is traveling. As another example, the position data may be used to obtain weather information about the location where the master AV 102 is located. The master AV 102 may also utilize RTK technology and may include a high precision RTK module.

[0154] The one or more sensors 134 represent at least one onboard vehicle sensor that may be mounted on the master AV 102 and may be used by the AV 102 for its autonomous operation and / or for obtaining data for testing or evaluating the master AV 102. The sensor(s) 134 may include a vehicle camera, a radar device, a lidar device, an ignition timing sensor, an exhaust sensor, a vehicle speed sensor, an accelerometer, a battery sensor, a parking assist sensor, a lane change and / or blind spot sensor, a lane assist sensor, a tire pressure sensor, a fluid level sensor (including a fuel level sensor), a brake pad wear sensor, and a rain or precipitation sensor. In general, the sensor(s) 134 are used to obtain vehicle sensor data, which may include vehicle sensor values ​​measured or determined by the sensors. The sensor data from the sensor(s) 134 may be captured during testing and then evaluated as part of evaluating the master AV 102 at the time of testing.

[0155] The onboard vehicle computer 130 is an onboard computer because it is carried by the host AV 102 and because it is part of the vehicle electronics 104, it is considered a vehicle computer. The onboard vehicle computer 130 includes at least one processor 142 and non-transitory computer-readable memory 144 accessible by the at least one processor 142. The onboard vehicle computer 130 is connected to a vehicle communication bus 136 and can use the bus 136 to send and receive messages to and from other vehicle components. The onboard vehicle computer 130 also includes short-range wireless communication (SRWC) or dedicated short-range communication (DSRC) circuitry 146 and a cellular chipset 148 for wireless communication. The DSRC circuitry 146 includes an antenna and is configured to execute one or more SRWC technologies, such as IEEE 802.11 protocols (e.g., IEEE 802.11p, Wi-Fi TM )、WiMAX TM 、ZigBee TM 、Z-Wave TM 、Wi-Fi direct TM 、Bluetooth TM (For example, Bluetooth TM In one embodiment, the DSRC circuit 146 may be used to perform communications with the backend server 112—for example, the master AV 102 may use the DSRC circuit 146 to send a message to the roadside unit (or units) 108, which may then forward the message to the backend server 112 via the land network 114 to which the RSU 108 is connected. The cellular chipset 148 includes an antenna and is used to perform cellular communications or long-range radio communications with the wireless carrier system 116. And, in one embodiment, the cellular chipset 148 includes suitable 5G hardware and 5G configurations so that 5G communications may be performed between the master AV 102 and the wireless carrier system 116, such as for the purpose of performing communications between the master AV 102 and one or more remote devices / computers (such as those implementing the backend server 112).

[0156] Any one or more of the processors discussed herein are electronic processors that can be implemented as any suitable electronic hardware capable of processing computer instructions and can be selected based on the application in which it will be used. Examples of processor types that can be used include central processing units (CPUs), graphics processing units (GPUs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), microprocessors, microcontrollers, etc. Any one or more of the memories or memory devices described herein are or include non-transient computer-readable memories and can be implemented as any suitable type of memory capable of storing data or information in a non-volatile manner and in electronic form so that the stored data or information can be used by the processor. The memory can be any of a variety of different electronic memory types and can be selected based on the application in which it will be used. Examples of memory types that can be used include disk or optical drive, ROM (read-only memory), solid-state drive (SSD) (including other solid-state storage devices, such as solid-state hybrid drive (SSHD)), other types of flash memory, hard disk drive (HDD), non-volatile random access memory (NVRAM), etc. It should be understood that any one or more computers discussed herein may include other memories, such as volatile RAM used by a processor and / or multiple processors.

[0157] Reference Fig.25 , an embodiment of a method 200 for safely testing AV is shown, and according to at least some embodiments, the method 200 is implemented as a method for safely testing AV in a natural environment and / or a method for naturally safely testing AV. The method 200 or one or more steps thereof (e.g., steps 210 and / or 220) may be performed by the AV test system 110. In such an embodiment, the AV test system 110 may include computer instructions stored on a memory, which, when executed by at least one processor, causes associated steps and / or functions to be performed, such as the steps in steps 210 and 220. According to at least some embodiments, the AV test system 110 is or includes a portion of the backend server 112, such as one or more processors and / or memories.

[0158] The method 200 begins at step 210, where a machine learning (ML) agent is generated. For example, the machine learning agent uses a policy modeled by an ML model (e.g., a neural network), which can be a deep reinforcement learning (DRL) agent. In the illustrated embodiment, generating the ML agent includes steps 212-216. In step 212, a test state model including critical states and non-critical states is obtained. In at least one embodiment, the test problem or environment is expressed or represented as a test state model, such as a sequential MDP, in which the manipulation of the background vehicle (BV) is determined based on the current state that encodes information about the BV and the host AV (or the AV under test) (e.g., position and velocity), in which embodiment, the MDP is the test state model. In other embodiments, the test state model can be another model of the environment, such as a non-Markov state-action model, which can be edited to remove non-critical states. The method 200 proceeds to step 214.

[0159] In step 214, the test state model is edited to omit or remove non-critical states. Non-critical states may refer to non-safety-critical states. In one embodiment, for example when using the disclosed D2RL method, the test state model (e.g., MDP) may be edited to remove non-critical (or non-safety-critical) states and reconnect critical (or safety-critical) states, e.g. Figure 3 As shown. This edited test state model is referred to as an edited test state model, and the edited MDP is referred to as an edited MDP. In some embodiments, for example, in the case of using the disclosed D2RL method, the step also includes or results in the densification of information used as training data for training the ML agent in step 216. At least according to some embodiments, the systems and methods provided herein solve the rarity disaster and / or dimensionality disaster as described above by implementing the disclosed D2RL method, in which the Markov decision process (or other test state model) is edited by removing non-critical states and reconnecting critical states to each other so that the use of information in the training data is densified. Therefore, in such an embodiment, the edited state test model may also be referred to as a densified state test model. Method 200 proceeds to step 216.

[0160] In step 216, a machine learning (ML) agent is trained using the edited state test model. In at least some embodiments, the ML agent is a deep reinforcement learning (DRL) agent or a dense DRL (D2RL) agent, such as when using the disclosed D2RL method. The ML agent employs an ML model, such as a neural network, which is trained using an edited or densified state test model. Since non-critical data is removed as part of removing non-critical states, the data used for training is densified relative to safety-critical data, so this can overcome or mitigate rare disasters. In some embodiments of the disclosed D2RL method, a background vehicle (BV) is trained in a natural driving environment simulator to learn when to perform a specific adversarial maneuver. For example, using the disclosed D2RL method, according to at least one embodiment, a BV is trained in a natural driving environment simulator to learn when to perform a specific adversarial maneuver, thereby generating an intelligent test environment that can maximize evaluation efficiency while ensuring unbiased estimation. As described above, at least according to some embodiments, this produces an intelligent test environment that can maximize evaluation efficiency while ensuring unbiased estimation. The method 200 continues to step 220 .

[0161] In step 220, one or more background vehicles are configured to operate according to the trained ML agent. The trained ML agent may be represented in the AV test system 110 and may be compiled into data, such as a set of parameters determined as a result of training, such as parameters of a neural network. In at least one embodiment, the trained ML agent is stored in the memory 152 of the backend server 112. In at least one embodiment where the (one or more) BVs are or include virtual BVs, the simulation to be used for testing may be configured to use the trained ML agent to select maneuvers and / or other operations for the (one or more) BVs during testing. In at least one embodiment where the (one or more) BVs are or include virtual BVs, the real (one or more) BVs may be configured to receive commands or instructions from the trained ML agent, which may be configured to run as part of the simulation at the backend server 112, the real BV, and / or another appropriate computing device (e.g., RSU 108). The method 200 continues to step 230.

[0162] In step 230, the host AV is tested in an environment including one or more background vehicles. The host AV is tested in a test environment, which is a predetermined or designated test location for testing the host AV, and when performing road testing, the test environment can be a closed track test facility or one or more roads. According to at least one embodiment, the host AV is tested in an augmented reality (AR) manner, wherein the host AV operates in the real world, such as in a real test environment (as opposed to a virtual test environment), and wherein virtual elements are introduced into the host AV environment by, for example, sending a virtual or simulated BSM from a roadside unit to the host AV, such as described in U.S. Patent Application Publication No. 2020 / 0065443. Such a simulation system that provides an AR environment for testing an AV is referred to as an AR AV testing system.

[0163] In one embodiment, a simulation system described in U.S. Patent Application Publication No. 2020 / 0065443 may be used. The simulation system provides simulated virtual objects for use by a connected real vehicle (here, a host AV) during testing of the vehicle on the road. The discussion of the simulation system in paragraphs

[0015] -

[0021] ,

[0047] -

[0095] of U.S. Patent Application Publication No. 2020 / 0065443 and the (one or more) methods discussed in paragraphs

[0004] -

[0014] ,

[0022] -

[0034] ,

[0047] -

[0049] and

[0096] -

[0106] of U.S. Patent Application Publication No. 2020 / 0065443 are incorporated herein by reference. Such a simulation system can be used to test the host AV in a real environment enhanced by a virtual vehicle (here, a background vehicle), and in particular, such a simulation system can be modified to include a BV that operates according to a trained ML agent. Thus, according to at least some embodiments, natural safety testing can be performed on the host AV in a real environment and in a manner that addresses the above-mentioned curse of rarity and curse of dimensionality. Method 200 then ends.

[0164] In other embodiments, the disclosed D2RL method, one or more steps of method 200, and / or other features described herein may be performed for a virtual master AV, a real or virtual vehicle, or other applications. It should be understood that, as described above, the disclosed D2RL method may be generally applicable to other vehicle testing or product testing that may typically present similar problems to the above-described problems of the curse of rarity and the curse of dimensionality.

[0165] It should be understood that the foregoing description is one or more embodiments of the present invention. The present invention is not limited to the specific embodiment(s) disclosed herein, but is limited only by the appended claims. In addition, the statements contained in the foregoing description are related to the disclosed embodiments and should not be interpreted as limitations on the scope of the present invention or on the definitions of the terms used in the claims, except where a term or phrase is clearly defined above. Various other embodiments of the disclosed embodiment(s) and various changes and modifications will become apparent to those skilled in the art.

[0166] As used in this specification and claims, when used in conjunction with a list of one or more components or other items, the terms "such as," "for example," "as," "such as," and "like," and the verbs "include," "have," "comprise," and their other verb forms, are each to be interpreted as open-ended, meaning that the list is not to be viewed as excluding other, additional components or items. Other terms are to be interpreted using their broadest reasonable meaning unless they are used in a context that requires a different interpretation. In addition, the term "and / or" is to be interpreted as an inclusive OR. Thus, for example, the phrase "A, B, and / or C" is to be interpreted to include all of: "A"; "B"; "C"; "A and B"; "A and C"; "B and C"; and "A, B, and C."

[0167] References

[0168] 1.LeCun,Y.,Bengio,Y.,&Hinton,G.(2015).Deep learning.

[0169] Nature, 521(7553), 436-444.

[0170] 2. Insider, 10 million self-driving cars will be on the road by 2020, https: / / www.businessinsider.com / report-10-million- self-driving-cars-will-be-on

[0171] the-road-by-2020-2015-5-6 ,2016.

[0172] 3. Nissan promises self-driving cars by

[0173] 2020, https: / / www.wired.com / 2013 / 08 / nissan-autonomous-drive / ,2014.

[0174] 4. Insider, Tesla's self-driving vehicles are not far off. https: / / www.businessinsider.com / elon-musk-on-teslas- autonomous-cars-2015-9 ,2015.

[0175] 5.Society of Automotive Engineers, Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles, https: / / www.sae.org / standards / content / j3016_202104 / ,2021.

[0176] 6. Kalra, N., & Paddock, SM (2016). Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability? Transportation Research Part A: Policy and Practice, 94, 182-193.

[0177] 7. California Department of Motor Vehicles, Disengagement reports, https: / / www.dmv.ca.gov / portal / vehicle- industry-services / autonomous-vehicles / d isengagement-reports / ,2020.

[0178] 8. Paz, D., Lai, P. J., Chan, N., Jiang, Y., & Christensen, H. I. (2020, September). Autonomous vehicle benchmarking using unbiased metrics. In 2020 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 6223-6228). IEEE.

[0179] 9. Favarò, F., Eurich, S., & Nader, N. (2018). Autonomous vehicles' disengagements: Trends, triggers, and regulatory limitations. Accident Analysis & Prevention, 110, 136-148.

[0180] 10. Donoho, DL (2000). High-dimensional data analysis: The curses and blessings of dimensionality. AMS math challenges lecture, 1 (2000), 32.

[0181] 11.Sutton, RS, & Barto, AG (2018). Reinforcement learning: An introduction. MIT press.

[0182] 12.Megahed, FM, Chen, YJ, Megahed, A., Ong, Y., Altman, N., & Krzywinski, M. (2021). The class imbalance problem. Nature Methods, 18(11), 1270-1272.

[0183] 13. Hinton, GE, & Salakhutdinov, RR (2006). Reducing the dimensionality of data with neural networks. Science, 313(5786), 504-507.

[0184] 14.Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., ... & Hassabis, D. (2017). Mastering the game of go without human knowledge. Nature, 550(7676), 354-359.

[0185] 15. Mirhoseini, A., Goldie, A., Yazgan, M., Jiang, J. W., Songhori, E., Wang, S., ... & Dean, J. (2021). A graph placement methodology for fast chip design. Nature, 594(7862), 207-212.

[0186] 16. Koren, M., Alsaif, S., Lee, R., & Kochenderfer, MJ (2018, June). Adaptive stress testing for autonomous vehicles. In 2018 IEEE Intelligent Vehicles Symposium (IV) (pp. 1-7). IEEE.

[0187] 17. Pek, C., Manzinger, S., Koschi, M., & Althoff, M. (2020). Using online verification to prevent autonomous vehicles from causing accidents. Nature Machine Intelligence, 2(9), 518-528.

[0188] 18. Katz, G., Barrett, C., Dill, D. L., Julian, K., & Kochenderfer, M. J. (2017, July). Reluplex: An efficient SMT solver for verifying deep neural networks. In International Conference on Computer Aided Verification (pp. 97-117). Springer, Cham.

[0189] 19. Feng, S., Feng, Y., Yu, C., Zhang, Y., HX Liu. (2020). Testing scenario library generation for connected and automated vehicles, Part I: Methodology. IEEE Transactions on Intelligent Transportation Systems, 22(3), 1573-1582.

[0190] 20. Feng, S., Y. Feng, H. Sun, S. Bao, Y. Zhang, H. X. Liu. (2020). Testing scenario library generation for connected and automated vehicles, Part II: Case studies. IEEE Transactions on Intelligent Transportation Systems, 22(9), 5635-5647.

[0191] 21. Feng, S., Feng, Y., Sun, H., Zhang, Y., & Liu, H.X. (2020). Testing scenario library generation for connected and automated vehicles: an adaptive framework. IEEE Transactions on Intelligent Transportation Systems, 23(2), 1213-1222.

[0192] 22. Feng, S., Yan, X., Sun, H., Feng, Y., & Liu, H. X. (2021). Intelligent driving intelligence test for autonomous vehicles with naturalistic and adversarial environments. Nature communications, 12(1), 1-14.

[0193] 23. Sinha, A., O'Kelly, M., Tedrake, R., & Duchi, J. C. (2020). Neural bridge sampling for evaluating safety-critical autonomous systems. Advances in Neural Information Processing Systems, 33.

[0194] 24.Li, L. et al. Parallel testing of vehicle intelligence via virtual-real interaction. Sci. Robot. 4, eaaw4106 (2019).

[0195] 25. Li, L., Zheng, N., & Wang, FY (2020). A theoretical foundation of intelligence testing and its application for intelligent vehicles. IEEE Transactions on Intelligent Transportation Systems, volume 22, issue 10, 6297-6306.

[0196] 26.Simulation City: Introducing Waymo's most advanced simulation system yet for autonomous driving. https: / / blog.waymo.com / 2021 / 06 / SimulationCity.html

[0197] 27. S. Kato, S. Tokunaga, Y. Maruyama, S. Maeda, M. Hirabayashi, Y. Kitsukawa, A. Monrroy, T. Ando, ​​Y. Fujii, and T. Azumi, ``Autoware on Board: Enabling Autonomous Vehicles with Embedded Systems, ” In Proceedings of the 9th ACM / IEEE International Conference on Cyber-Physical Systems (ICCPS 2018), pp. 287-296, 2018.

[0198] 28. Feng, S., Feng, Y., Yan, X., Shen, S., Xu, S., & Liu, H. X. (2020). Safety assessment of highly automated driving systems in test tracks: a new framework. Accident Analysis & Prevention, 144, 105664.

[0199] 29.Lopez,PA,Behrisch,M.,Bieker-Walz,L.,Erdmann,J., YP,Hilbrich,R.,...&Wieβner,E.(2018,November).Microscopic traffic simulation using sumo.In 2018 21st International Conference on Intelligent Transportation Systems (ITSC) (pp.2575-2582).IEEE.

[0200] 30. D. Bezzina, J. Sayer, Safety pilot model deployment: Test conductor team report. (Report No. DOT HS 812 171). Washington, DC: National Highway Traffic Safety Administration (2014).

[0201] 31. J. Sayer, D. LeBlanc, S. Bogard, D. Funkhouser, S. Bao, M. L. Buonarosa, A. Blankespoor, Integrated Vehicle-Based Safety Systems Field Operational Test: Final Program Report (No. FHWA-JPO-11-150; UMTRI-2010-36). United States. Joint Program Office for Intelligent Transportation Systems (2011).

[0202] 32. Arun, A., Haque, M. M., Bhaskar, A., Washington, S., & Sayed, T. (2021). A systematic mapping review of surrogate safety assessment using traffic conflict techniques. Accident Analysis & Prevention, 153, 106016.

[0203] 33. Weng, B., Rao, S. J., Deosthale, E., Schnelle, S., & Barickman, F. (2020, June). Model predictive instantaneous safety metric for evaluation of automated driving systems. In 2020 IEEE Intelligent Vehicles Symposium (IV) (pp. 1899-1906). IEEE.

[0204] 34. Junietz, P., Bonakdar, F., Klamann, B., & Winner, H. (2018, November). Criticality metric for the safety validation of automated driving using model predictive trajectory optimization. In 2018 21st International Conference on Intelligent Transportation Systems (ITSC) (pp. 60-65). IEEE.

[0205] 35. Sun, H., Feng, S., Yan, X., & Liu, H. X. (2021). Corner Case Generation and Analysis for Safety Assessment of Autonomous Vehicles. Transportation Research Record. DOI: 10.1177 / 03611981211018697.

[0206] 36. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy optimization algorithms. https: / / arxiv.org / abs / 1707.06347 .

[0207] 37. Owen, AB Monte Carlo Theory, Methods and Examples (Monte Carlo theory, methods and examples). https: / / statweb.stanford.edu / ~owen / mc / (2013).

[0208] 38. Liang, E. et al. RLlib: Abstractions for Distributed Reinforcement Learning (2018). https: / / arxiv.org / abs / 1712.09381 .

[0209] 39. Treiber, M., Hennecke, A. & Helbing, D. Congested traffic states inempirical observations and microscopic simulations. Phys. Rev. E 62, 1805 (2000).

[0210] 40. Kesting, A., Treiber, M. & Helbing, D. General lane-changing model MOBIL for car-following models. Transport. Res. Rec. 1999, 86–94 (2007).

[0211] 41. Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017). Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4700-4708).

[0212] 42. Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009, June). Curriculum learning. In Proceedings of the 26th annual international conference on machine learning (pp. 41-48).

[0213] 43.Au, SK, & Beck, JL (2003). Important sampling in high dimensions. Structural safety, 25(2), 139-163.

[0214] 44.Silver, D., Singh, S., Precup, D., & Sutton, RS (2021). Reward is enough. Artificial Intelligence, 103535.

[0215] 45. Yan, X., Feng, S., Sun, H., & Liu, H. X. (2021). Distributionally Consistent Simulation of Naturalistic Driving Environment for Autonomous Vehicle Testing. https: / / arxiv.org / abs / 2101.02828 .

[0216] 46. ​​Chang AX, Funkhouser T, Guibas L, Hanrahan P, Huang Q, Li Z, Savarese S, Savva M, Song S, Su H, Xiao J. Shapenet: An information-rich 3D model repository. arXiv preprint arXiv:1512.03012.2015Dec 9 (December 9, 2015).

[0217] 47. Darweesh, H., Takeuchi, E., Takeda, K., Ninomiya, Y., Sujiwo, A., Morales, LY, ... & Kato, S. (2017). Open source integrated planner for autonomous navigation in highly dynamic environments. Journal of Robotics and Mechatronics, 29(4), 668-684.

Claims

1. A method for safely testing a primary autonomous vehicle (AV), comprising the following steps: Generating a trained machine learning (ML) agent by the following steps: (i) obtaining a test state model having a non-safety critical state and a safety critical state, (ii) editing the test state model to obtain an edited test state model omitting data about the non-safety critical state, and (iii) training an ML agent using the edited state test model to generate a trained ML agent; as well as The host AV is tested in an environment including one or more background vehicles configured to operate according to the trained ML agent.

2. The method of claim 1 , further comprising the step of configuring the one or more background vehicles to operate in accordance with the trained ML agent, wherein the configuring step comprises storing data representing the trained ML agent in a computer-readable memory.

3. The method of claim 1, wherein the trained ML agent is a deep reinforcement learning (DRL) agent, and wherein the trained ML agent employs a neural network.

4. The method of claim 3, wherein the trained ML agent is a dense DRL (D2RL) agent.

5. The method according to claim 4, wherein the test state model is a Markov decision process (MDP) or is based on a Markov decision process (MDP).

6. The method of claim 5, wherein a D2RL approach is used to densify safety-critical data used to train the ML agent.

7. The method of claim 5, wherein the editing sub-step includes removing the non-safety critical state and reconnecting the safety critical state.

8. The method of claim 1 , wherein the environment in which the host AV is tested is a real environment having one or more roads on which the host AV travels during testing, and wherein the one or more background vehicles are virtual vehicles or simulated vehicles.

9. The method of claim 8, wherein the testing step includes performing a simulation synchronized with the host AV and an environment in which the host AV is tested, and wherein the simulation includes at least one of the one or more background vehicles as a virtual background vehicle.

10. The method of claim 1, wherein the method is performed by an augmented reality (AR) autonomous vehicle (AV) testing system.

11. An autonomous vehicle (AV) testing system comprising: at least one electronic processor and a memory accessible by the at least one electronic processor, wherein the memory stores computer instructions; wherein the AV testing system is configured such that when the at least one electronic processor executes the computer instructions, the AV testing system: Generating a trained machine learning (ML) agent by the following steps: (i) obtaining a test state model having a non-safety critical state and a safety critical state, (ii) editing the test state model to obtain an edited test state model omitting data about the non-safety critical state, and (iii) training an ML agent using the edited state test model to generate a trained ML agent; as well as The host AV is tested in an environment including one or more background vehicles configured to operate according to the trained ML agent.

12. The AV testing system of claim 11, wherein the AV testing system is configured such that when the at least one electronic processor executes the computer instructions, the AV testing system: configures the one or more background vehicles to operate in accordance with the trained ML agent, wherein the configuring step includes storing data representing the trained ML agent in a computer-readable memory.

13. The AV testing system of claim 11, wherein the trained ML agent is a deep reinforcement learning (DRL) agent, and wherein the trained ML agent employs a neural network.

14. The AV testing system of claim 13, wherein the trained ML agent is a dense DRL (D2RL) agent.

15. The AV testing system of claim 14, wherein the test state model is a Markov decision process (MDP), or is based on a Markov decision process (MDP).

16. The AV testing system of claim 15, wherein a D2RL approach is used to densify safety-critical data used to train the ML agent.

17. The AV testing system of claim 15, wherein the editing sub-step includes removing the non-safety critical state and reconnecting the safety critical state.

18. The AV testing system of claim 11, wherein the environment in which the host AV is tested is a real environment having one or more roads on which the host AV travels during the test, and wherein the one or more background vehicles are virtual vehicles or simulated vehicles.

19. The AV testing system of claim 18, wherein the testing step includes performing a simulation synchronized with the host AV and an environment in which the host AV is tested, and wherein the simulation includes at least one of the one or more background vehicles as a virtual background vehicle.

20. A method for safely testing a primary autonomous vehicle (AV), comprising the steps of: Generating a trained dense deep reinforcement learning (D2RL) agent by: (i) obtaining a test state model having non-safety critical states and safety critical states, (ii) editing the test state model to omit at least one non-safety critical state and reconnect at least two safety critical states, and (iii) training a D2RL agent using the edited state test model to generate a trained D2RL agent; configuring one or more background vehicles to operate according to the trained D2RL agent; as well as After configuring one or more background vehicles to operate according to the trained D2RL agent, the AV is tested in an environment including the one or more background vehicles.

Images