SIMULATING TRAFFIC SCENARIOS

Abstract

A computer comprising a processor and memory, the memory containing instructions executable by the processor to generate initial traffic scenarios based on initial object locations and trajectories of one or more objects, the initial traffic scenarios being generated using a three-dimensional simulation engine. The probabilities of a collision between the one or more objects included in the initial traffic scenarios can be determined. Select a subset of the initial traffic scenarios containing collision probabilities greater than a user-defined threshold. Second traffic scenarios can be generated by perturbing the selected subset, using the 3D simulation engine. A machine learning system can then be trained on both the first and second traffic scenarios.

Description

AREA OF TECHNOLOGY

[0001] This revelation concerns the simulation of traffic scenarios. GENERAL STATE OF THE ART

[0002] Computers can operate systems and / or devices, including vehicles, robots, drones, and / or object tracking systems. Data, including images, can be captured by sensors and processed by a computer to determine the location of a system relative to its environment and to objects within that environment. A computer can use this location data to determine one or more trajectories for operating the system or its components within the environment. SUMMARY

[0003] Systems, including vehicles, robots, drones, etc., can be operated by acquiring sensor data, including data about the environment surrounding the system, and processing this sensor data to determine the locations of objects in the environment. This determined location data could then be processed to determine the operation of the system or parts of the system. For example, a robot could determine the position of the arm of another nearby robot. The robot could then use this determined robot arm position to determine a path for its gripper to move in order to grasp a workpiece without colliding with the other robot's arm. In another example, a vehicle could determine the location of another vehicle traveling on a roadway.The vehicle could use the determined location of the other vehicle to determine a path along which to operate, while maintaining a predetermined distance from the other vehicle. Vehicle operation is used as a non-restrictive example of system location determination in the description below.

[0004] In examples in this document, a vehicle can use a machine learning system, such as a convolutional neural network, to determine the identities and locations of one or more objects in its environment, such as other vehicles, pedestrians, and obstacles, including traffic barriers, signage, etc. Training a machine learning system, such as a convolutional neural network, to identify and locate objects may require a training dataset containing thousands of video sequences, which in turn may contain millions of frames. Additionally, training a machine learning system, such as a convolutional neural network, may require ground-truth data for the images in the training dataset.The ground truth includes commentary data regarding the identities and locations of objects contained in the training dataset, which were collected from a source other than the machine learning system.

[0005] A training dataset can be generated by capturing video sequences of real vehicles driving on real roads. The captured video sequences can be annotated by having people view them and manually enter the identities and locations of objects appearing in the video. Training a machine learning system can benefit from training datasets that include video sequences containing collisions and near misses between the vehicle capturing the video and objects in the environment. A near miss occurs when the vehicle capturing the video comes within a user-defined distance, for example, one meter, of an object while in operation.Training a machine learning system with video sequences that include impacts and near misses can teach the machine learning system to predict impacts and near misses based on captured sensor data and to warn the vehicle computer so that it can take appropriate action.

[0006] Capturing video sequences of real vehicles that include impacts and near misses can be difficult and infrequent. Traffic scenario simulation techniques, as described in this paper, can improve the training of machine learning systems by generating simulated traffic scenarios based on computer simulations of traffic scenes and objects. A traffic scenario is a video sequence that simulates the view from the perspective of a vehicle sensor, rendered in a photorealistic manner. Photorealistic means that a simulated image can appear to a machine learning system as if it were captured by a real sensor; for example, a machine learning system would achieve the same result if it were fed both a real image of a scene and a photorealistically simulated version of the same scene.

[0007] Traffic scenario simulation techniques, as described in this paper, can improve the training of machine learning systems by photorealistically simulating a large number of traffic scenarios and evaluating them to determine which scenarios include collisions or near misses. Simulated traffic scenarios containing collisions and near misses can then be perturbed to generate additional collision or near miss scenarios. Perturbing traffic scenarios involves slightly modifying the initial conditions used to generate them, resulting in multiple variations of the scenarios that include collisions and near misses.Generating multiple variations of traffic scenarios in this way improves the training of the machine learning system by providing multiple variations of impact and near-impact scenarios and increasing the probability that the machine learning system will correctly predict impact and near-impact scenarios.

[0008] This document discloses a method comprising: generating initial traffic scenarios involving one or more objects based on initialization and setup data for the one or more objects, wherein the initial traffic scenarios are generated using a three-dimensional simulation engine; and determining the probability of a collision between the one or more objects included in the initial traffic scenarios. A subset of the initial traffic scenarios can be selected that includes a collision probability greater than a user-selected threshold.Second traffic scenarios can be generated by perturbing a selected subset of the first traffic scenarios. These second traffic scenarios are generated using the 3D simulation engine, and a machine learning system can be trained based on both the first and second traffic scenarios. The selected subset of the first traffic scenarios can be determined by modifying one or more parameters contained in the initialization and setup data of the one or more objects. The 3D simulation engine can receive the initialization and setup data and determine simulated video sequences that include object motion trajectories for the one or more objects. The simulated video sequences can be based on photorealistic image rendering. The object motion trajectories for the one or more objects can be based on physical simulation.

[0009] The initialization and setup data can include initial locations and trajectories for one or more objects. The probability of an impact for one or more objects can be based on four impact metrics and a random forest regressor. The four impact metrics can be based on longitudinal deceleration, maximum permissible longitudinal deceleration, longitudinal acceleration, maximum permissible longitudinal acceleration, lateral acceleration, and maximum permissible lateral acceleration. The machine learning system can be a neural network that includes convolutional and fully connected layers. The trained machine learning system can be transferred to a system incorporated into a vehicle. The vehicle can then be operated based on the trained machine learning system.The vehicle can be operated by controlling one or more of its propulsion, steering, and braking systems. The random forest regressor can include multiple decision trees. These decision trees can randomly sample the four impact metrics across multiple traffic scenarios.

[0010] Furthermore, a computer-readable medium is disclosed that stores program instructions for executing some or all of the preceding procedure steps. A computer is also disclosed that is programmed to execute some or all of the preceding procedure steps and includes a computing device programmed to generate initial traffic scenarios involving one or more objects based on initialization and setup data for the one or more objects, wherein the initial traffic scenarios are generated using a three-dimensional simulation engine, and to determine a probability of a collision between the one or more objects included in the initial traffic scenarios. A subset of the initial traffic scenarios can be selected that includes a collision probability greater than a user-selected threshold.Second traffic scenarios can be generated by perturbing a selected subset of the first traffic scenarios. These second traffic scenarios are generated using the 3D simulation engine, and a machine learning system can be trained based on both the first and second traffic scenarios. The selected subset of the first traffic scenarios can be determined by modifying one or more parameters contained in the initialization and setup data of the one or more objects. The 3D simulation engine can receive the initialization and setup data and determine simulated video sequences that include object motion trajectories for the one or more objects. The simulated video sequences can be based on photorealistic image rendering. The object motion trajectories for the one or more objects can be based on physical simulation.

[0011] The instructions can include further instructions, with the initialization and setup data potentially including initial locations and trajectories for the one or more objects. The probability of an impact for the one or more objects can be based on four impact metrics and a random forest regressor. The four impact metrics can be based on longitudinal deceleration, maximum permissible longitudinal deceleration, longitudinal acceleration, maximum permissible longitudinal acceleration, lateral acceleration, and maximum permissible lateral acceleration. The machine learning system can be a neural network incorporating convolutional and fully connected layers. The trained machine learning system can be deployed to a system integrated into a vehicle. The vehicle can then be operated based on the trained machine learning system.The vehicle can be operated by controlling one or more of its propulsion, steering, and braking systems. The random forest regressor can include multiple decision trees. These decision trees can randomly sample the four impact metrics across multiple traffic scenarios. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 is a block diagram of an example vehicle system. Fig. 2 is a presentation of exemplary data on traffic scenarios. Fig. Figure 3 is a representation of an exemplary image of a traffic scenario. Fig. Figure 4 is a representation of an exemplary system for generating traffic scenarios. Fig. Figure 5 is a representation of an exemplary system for generating additional traffic scenarios. Fig. Figure 6 is a flowchart of an exemplary process for generating traffic scenarios. Fig. Figure 7 is a flowchart of an exemplary process for generating additional traffic scenarios. Fig. Figure 8 is a flowchart of an example process for operating a vehicle. DETAILED DESCRIPTION

[0012] Fig. Figure 1 is a representation of a vehicle computer system 100. The vehicle computer system 100 includes a vehicle 110, a computing device 115 located in the vehicle 110, and a server computer 120 located remotely from the vehicle 110. One or more computing devices 115 of the vehicle 110 can receive data relating to the operation of the vehicle 110 from sensors 116. The computing device 115 can operate the vehicle 110 based on data received from the sensors 116 and / or data received from the remote server computer 120. The server computer 120 can communicate with the vehicle 110 via a network 130.

[0013] The computing device 115 includes a processor and a memory, as known. Furthermore, the memory includes one or more forms of computer-readable media and stores instructions that can be executed by the processor to perform various operations, including those disclosed herein. For example, the computing device 115 may include programming to operate one or more of the vehicle's brakes, drive (i.e., controlling acceleration in the vehicle 110 by controlling one or more internal combustion engines, electric motors, hybrid motors, etc.), steering, climate control, interior and / or exterior lights, etc., and to determine whether and when the computing device 115 should control such operations as opposed to a human operator.

[0014] The computing device 115 can include more than one computing device, i.e., controllers or the like, which are incorporated in the vehicle 110 for monitoring and / or controlling various vehicle components, i.e., a drive control 112, a brake control 113, a steering control 114, etc., or be communicatively coupled to it, i.e., via a vehicle communication bus, as described in more detail below. The computing device 115 is generally arranged for communication on a vehicle communication network, i.e., which includes a bus in the vehicle 110, such as a Controller Area Network (CAN) or the like; the vehicle 110 network can additionally or alternatively include wired or wireless communication mechanisms of a kind known to exist, i.e., Ethernet or other communication protocols.

[0015] The computing device 115 can transmit messages to various devices in the vehicle 110 via the vehicle network and / or receive messages from the various devices, i.e., controllers, actuators, sensors, etc., including the sensors 116. Alternatively or additionally, in cases where the computing device 115 actually comprises several devices, the vehicle communication network can be used for communication between devices that are referred to in this disclosure as the computing device 115. Furthermore, as mentioned below, various controllers or sensing elements, such as the sensors 116, can provide data to the computing device 115 via the vehicle communication network.

[0016] Additionally, the computing device 115 can be configured to communicate with a remote server computer 120, i.e., a cloud server, via a network 130, which, as described below, includes hardware, firmware, and software that enable the computing device 115 to communicate with a remote server computer 120 via a network 130, such as wireless internet (WI-FI®) or cellular networks, through a vehicle-to-infrastructure (V2I) interface 111. The V2X interface 111 can accordingly include processors, memory, transceivers, etc., configured to utilize various wired and / or wireless network technologies, i.e., cellular, BLUETOOTH®, Bluetooth Low Energy (BLE), ultra-wideband (UWB), peer-to-peer communication, UWB-based radar, IEEE 802.11 and / or other wired and / or wireless packet networks or technologies.The computing device 115 can be configured to communicate with other vehicles 110 via a V2X (vehicle-to-everything) interface 111 using vehicle-to-vehicle (VV) networks, e.g., according to cellular communications (C-V2X), wireless cellular communications, dedicated short-range communications (DSRC), and / or the like, which are formed, for example, ad hoc between vehicles 110 in close proximity or via infrastructure-based networks. The computing device 115 also includes non-volatile memory of a type known to the public. The computing device 115 can log data in non-volatile memory for later retrieval and transmission via the vehicle communication network and a vehicle infrastructure (V2I) interface 111 to a server computer 120 or a mobile user device 160.

[0017] As already mentioned, instructions stored in memory and executable by the processor of the computing device 115 generally include programming to operate one or more components of the vehicle 110, i.e., braking, steering, propulsion, etc., without human intervention. Using data received by the computing device 115, i.e., sensor data from the sensors 116, the server computer 120, etc., the computing device 115 can make various decisions and / or control various components and / or operations of the vehicle 110. For example, the computing device 115 can program the operational behavior of the vehicle 110 (i.e., physical manifestations of the vehicle 110's operation), such as speed, acceleration, deceleration, steering, etc., and its tactical behavior (i.e.,Control of operational behavior in a manner typically intended to achieve efficient route driving), such as distance between vehicles and / or time intervals between vehicles, lane changes, minimum distance between vehicles, minimum time to cross the road when turning left, time to arrive at a specific location, and minimum time to cross the intersection at an intersection (without traffic lights).

[0018] Controllers, as used in this document, include computing devices that are typically programmed to monitor and / or control a specific vehicle subsystem. Examples include a drive controller 112, a brake controller 113, and a steering controller 114. A controller may be an electronic control unit (ECU), as known, which may include additional programming as described in this document. The controllers may be communicatively connected to the computing device 115 and receive instructions from it to operate the subsystem according to those instructions. For example, the brake controller 113 may receive instructions from the computing device 115 to operate the vehicle's brakes 110.

[0019] The one or more controllers 112, 113, 114 for the vehicle 110 may include known electronic control units (ECUs) or the like, which, as non-limiting examples, include one or more drive controllers 112, one or more brake controllers 113, and one or more steering controllers 114. Each of the controllers 112, 113, 114 may include respective processors and memory and one or more actuators. The controllers 112, 113, 114 may be programmed and connected to a communication bus of the vehicle 110, such as a Controller Area Network (CAN) bus or a Local Interconnect Network (LIN) bus, to receive instructions from the computing device 115 and to control actuators based on these instructions.

[0020] The sensors 116 can include a variety of devices known for providing data via the vehicle communication bus. For example, a radar mounted on a front bumper (not shown) of the vehicle 110 can provide the distance from the vehicle 110 to the next vehicle ahead, or a sensor for a global positioning system (GPS) located in the vehicle 110 can provide the vehicle 110's geographic coordinates. The distance(s) provided by the radar and / or the other sensors 116 and / or the geographic coordinates provided by the GPS sensor can be used by the computing device 115 to operate the vehicle 110, for example, autonomously or semi-autonomously.

[0021] The vehicle 110 is generally a land vehicle 110 capable of autonomous and / or semi-autonomous operation and has three or more wheels, i.e., a passenger car, a light truck, etc. The vehicle 110 includes one or more sensors 116, the V2I interface 111, the computing device 115, and one or more controllers 112, 113, 114. The sensors 116 can collect data relating to the vehicle 110 and the environment in which the vehicle 110 operates. By way of example, and not limited to, the sensors 116 can include, for example, altimeters, cameras, LiDAR, radar, ultrasonic sensors, infrared sensors, pressure sensors, accelerometers, gyroscopes, temperature sensors, Hall sensors, optical sensors, voltage sensors, current sensors, mechanical sensors such as switches, etc. The sensors 116 can be used to detect the environment in which the vehicle 110 is operated, i.e.The sensors 116 can detect phenomena such as weather conditions (precipitation, ambient temperature, etc.), the gradient of a road, the location of a road (i.e., using road edges, lane markings, etc.), or the locations of target objects, such as neighboring vehicles 110. The sensors 116 can also be used to collect data that includes dynamic data about the vehicle 110 relating to its operations, such as speed, yaw rate, steering angle, engine speed, brake pressure, oil pressure, the power level applied to the controls 112, 113, 114 in the vehicle 110, connectivity between components, and the precise and current performance of the vehicle 110's components.

[0022] The server computer 120 typically has features that it shares with the V2I interface 111 of the vehicle 110 and the computing device 115, such as a computer processor and memory, and a configuration for communication over a network 130. These features are therefore not described in detail to avoid redundancy. A server computer 120 can be used to develop and train software that can be transferred to a computing device 115 in a vehicle 110.

[0023] Fig. Section 2 presents exemplary initialization and setup data for generating traffic scenarios. In this document, initialization and setup data refers to data provided for generating a traffic scenario. This data includes parameters describing the background and objects to be rendered by photorealistic rendering software, including the initial locations and trajectories of the objects, and data describing the environmental conditions to be rendered. The initialization and setup data is provided to photorealistic rendering software to enable it to generate a video sequence containing video data frames, such as 2D images that appear as if captured by sensors 116 located in a vehicle 110.Parameters included in the initialization and setup data include a Map 200. The Map 200 is a set of digital data containing information that describes physical features at locations or coordinates, such as roadways 202, lane markings 204, traffic signals 218, vegetation 206 (including trees and bushes), and buildings 208. The Map 200 data can be used by photorealistic rendering software to generate background sections of images in a video sequence. Background sections can be portions of an image that do not contain moving objects, such as vehicles and pedestrians.Map 200 can contain three-dimensional (3D) data relating to the vegetation 206 and the buildings 208 to enable the photorealistic rendering software to render a video sequence containing two-dimensional (2D) views of Map 200. These views are rendered to appear as if captured by a video camera located in an ego-vehicle 210 on Map 200. The ego-vehicle 210 is so named because it represents the perspective from which the photorealistic rendering software generates images. The initialization and setup data can include the makes, models, and colors of vehicles included in the simulation to enable photorealistic rendering of the vehicles. The initialization and setup data also includes other objects, in this example, a second vehicle 214.

[0024] Parameters included in the initialization and setup data can also include initial locations and trajectories 212, 216 for the ego vehicle 210 and one or more second vehicles 214, respectively. The trajectories 212, 216 contain speed and direction data for the ego vehicle 210 and the second vehicle 214 at the respective times. The initialization and setup data can also include driving styles for the ego vehicle 210 and the second vehicle 214, including lateral and longitudinal accelerations as a function of time. The initial locations and trajectories 212, 216 can, for example, be assigned randomly. Other types of initialization and setup data include speed limits for lanes 202 and operating parameters of the ego vehicle 210 and the second vehicle 214, such as braking distances and following distances.The initialization and setup data may also include environmental conditions, such as lighting parameters like time of day, cloud cover, sun position, precipitation including rain, snow, sleet, etc., and other environmental conditions such as road friction values.

[0025] The initialization and setup data can also include data used by the photorealistic rendering software. This data can include the number of frames per second (FPS) in the rendered video sequence and the total number of seconds to be rendered. The initialization and setup data can also include the warm-up time. Warm-up time is the number of video frames required by the photorealistic rendering software to reach a stable state when rendering moving objects.

[0026] Fig. Figure 3 is a representation of a frame of video data 300, rendered by photorealistic rendering software as part of a sequence of video frames. The video data frame 300 can be rendered in response to initialization and setup data received by the photorealistic rendering software. An example of photorealistic rendering software is the Unreal Engine, provided by Epic Games, Inc., Cary, NC 27518. The video data frame 300 includes renderings of a roadway 302, lane markings 312, a traffic signal 314, buildings 304, vegetation 306, and a rendered vehicle 308, all rendered from the perspective of a video camera included in the ego-vehicle 210. The frame of video data 300 also includes an environmental condition, in this example light rain 310 (dashed lines).

[0027] Photorealistic rendering software can include a physics engine that simulates the motion of moving objects. A physics engine can contain a mathematical model for objects, such as vehicles, that includes their mass and lateral and longitudinal acceleration rates, enabling the photorealistic rendering software to realistically simulate their movement. Photorealistic rendering software can simulate the movement of objects, such as vehicles, by rendering each video data frame in the video sequence on a frame-by-frame basis. Each frame can be rendered to reflect changes in the objects' positions, while the scene is rendered using ray tracing to accurately simulate the effects of light rays as they are reflected from the objects onto a simulated camera lens.In this way, a video sequence simulating image data would be captured by a video camera included in an Ego Vehicle 210 when the Ego Vehicle 210 is driving on a map 200.

[0028] A video sequence generated by photorealistic rendering software can be realistic enough to be used for training a machine learning system. For example, a simulated video sequence can be generated by photorealistic rendering software that simulates a real video sequence captured by sensors 116 located in a vehicle 110 driving in the real world. By receiving the simulated video sequence, a trained machine learning system can determine the locations of objects within the simulated video sequence, achieving the same results as receiving the real video sequence.

[0029] The techniques described in this document for generating simulated video sequences offer several advantages over capturing real-world video sequences. Simulated video sequences can be generated much faster with fewer resources than real-world video sequences. Different weather conditions can be simulated without driving or waiting for seasonal changes. Simulated video sequences include ground-truth data regarding the locations of objects in the scene and require no further processing to be included in a training dataset. Because the physics engine included in photorealistic rendering software determines the locations of each object in the scene before rendering, data regarding the locations of objects in the scene is available for every frame of rendered data.The generated video sequence is also repeatable, meaning that simulating a second video sequence based on the same data will produce the same video sequence data as the first video sequence, unlike trying to repeat real video sequences.

[0030] The techniques described in this document for generating simulated video sequences have the advantage of being able to simulate collision or near-collision traffic scenarios without requiring real-world collision or near-collision scenarios to occur or be executed. Simulated traffic scenarios can be processed by software, as described in this document, to determine the probabilities of a collision or near-collision between objects included in the traffic scenarios.Techniques described in this paper can generate multiple variations of traffic scenarios and automatically evaluate the probability of a collision or near-collision to provide the training dataset with multiple variations of multiple collision and near-collision traffic scenarios in order to improve the ability of the machine learning system to correctly predict collision and near-collision traffic scenarios as they develop in real-world situations.

[0031] Fig. Figure 4 is a representation of a System 400 for investigating simulated traffic scenarios. The System 400 for investigating simulated traffic scenarios investigates the space of all traffic scenarios by generating simulated traffic scenarios based on randomized initialization and setup data 402. The initialization and setup data includes data as described above. The installation and setup data can be generated by analyzing a previously captured real-world video sequence. The installation and setup data can also be created from user input, where a background section of an image can be provided from map data, the objects, such as vehicles, can be specified from computer-aided design drawings, and users can input the initial locations and trajectories for the objects.Environmental conditions can be predefined from a library of images that includes a variety of lighting and atmospheric effects. Before generating the traffic scenario, the vehicle's location and trajectory data are randomly modified, typically within a predefined range, such as user-defined values, to provide variation in the generated traffic scenarios using a 404 randomizer. By randomizing the initialization and setup data in this way, a single set of initialization and setup data can be used to generate multiple traffic scenarios that exhibit variations in the paths taken by vehicles within those scenarios.

[0032] The randomizer 404 passes randomized initialization and setup data to the data store 406, which stores the randomized initialization and setup data in a memory included in the system 400 for investigating simulated traffic scenarios. The data store 406 forwards the randomized initialization and setup data to the photorealistic rendering software 408 to generate a traffic scenario that includes a video sequence and ground truth data, as described above in relation to Fig. 3 described. The rendered traffic scenario is passed to impact metrics 410 to determine whether the rendered traffic scenario involves a collision or near-collision between objects, in this example vehicles.

[0033] The impact metrics 410 determine a probability of an impact based on data obtained by analyzing traffic scenario data regarding the movement of the ego-vehicle 210 in the video sequence. The movement of the ego-vehicle 210 can be determined based on data regarding its location in successive frames of the video sequence. Location data can be extracted from the ground truth data output by the photorealistic rendering software and incorporated into the traffic scenario. In other examples, the video sequence can be processed by a machine learning system, such as a convolutional neural network, to determine the location of the ego-vehicle 210 in successive frames using visual odometry.The location data from successive video data frames can be used to determine a lateral and longitudinal acceleration for the Ego vehicle 210.

[0034] Lateral and longitudinal acceleration can be used to determine impact metrics, which can be combined to calculate the probability of a collision or near-collision for an Ego-Vehicle 210 based on a rendered traffic scenario. In this document, an "impact metric" refers to a numerical value that specifies the probability of a collision or near-collision based on measurements of object motion from a traffic scenario. Object motion can be determined, for example, by locating objects in successive video data frames.

[0035] The first impact metric, Rt1, Frame t is determined by the following equation: Rt1=max{adecel,tamdaaccel,tama

[0036] Where a decel, t the observed longitudinal deceleration at frame t is and a md the maximum permissible longitudinal deceleration. a md is determined by the user as the maximum permissible longitudinal deceleration above which the frame can be stopped by setting Rt1=100 = 100 is marked as a near miss. a accel,t is the observed longitudinal acceleration at frame t and a ma is the maximum permissible longitudinal acceleration. a ma is determined by the user as the maximum permissible longitudinal acceleration above which the frame can be set by adjusting Rt1=100 is marked as a near miss. The value of Rt1 is determined by the maximum of adecel,tamdund aaccel,tama is used in any frame.

[0037] The second impact metric, Rt2, Frame t is determined by the following equation: Rt2=(|alat,x|aml)

[0038] Where |a lat,x | the observed lateral acceleration is measured in the x-direction perpendicular to the direction of travel, and a ml the maximum permissible lateral acceleration. a ml The user determines the maximum permissible lateral acceleration above which the frame is marked as a near miss by setting the following: Rt2=100.

[0039] The third impact metric, Rt3, Frame t is determined by the following equation: Rt3=0.8∗Rt1+0.2∗Rt22

[0040] The third impact metric, Rt3, is the weighted average of Rt1 and Rt2.

[0041] The fourth impact metric, Rt4, Frame t is determined by the following equation: Rt4=∑t1(adecel,t>0.3g)∑t1(adecel,t>0.1g)∗0.5+∑t1(alat,x>0.4g)∑t1(alat,x>0.2g)∗0.5 Rt4 calculated how long the Ego vehicle 210 will operate with a longitudinal deceleration above 0.3 g relative to a longitudinal deceleration above 0.1 g, where g is an acceleration or deceleration equal to an gravitational force of one or 9.81 meters / second 2 is. Rt4 The program simultaneously calculates how long the ego vehicle 210 operates with a lateral acceleration above 0.4 g relative to a lateral acceleration of 0.2 g. The average of these two lateral acceleration ratios is a measure of the probability of a collision based on the operating behavior of the ego vehicle 210 at frame t of the traffic scenario.

[0042] The fifth impact metric, RT5 The impact metrics for frame t are determined using a random forest regressor. A random forest regressor is an estimation unit that fits multiple decision trees to subsamples of a dataset and uses averaging to improve the accuracy of the estimate. In this example, the decision trees are based on the magnitudes of the lateral and longitudinal accelerations. Equations (1)-(4) above can be expressed as a tree, with decision nodes based on user-defined ranges of values ​​for the input acceleration values. The end leaves of the trees are the impact metrics output by equations (1)-(4). The random forest regressor can calculate a mean of the impact metrics to estimate the probability of an impact. RT5 to determine the values ​​of the metrics for a traffic scenario. Rt1, Rt2, Rt3 and Rt4 Several traffic scenarios are randomly sampled to form multiple decision trees that provide a final probability of a collision. RT5 to predict. The Random Forest Regressor is used with a base probability. Rts, The training system uses a base probability, where the base probability in this example is the probability of a collision in a traffic scenario, such as "collision," "near collision," or "no collision," from traffic scenarios in the training dataset. Base probabilities can be determined by measuring the distance between objects in traffic scenarios within the training dataset. For example, the base probability can be... Rts as Rts=100 percent be coded when an impact is observed, as Rts=50 percent, when a near-collision is observed, and as Rts=0 percent when no impact or near-impact is observed. The Random Forest regressor is trained to calculate an impact metric. RT5 to spend that correspond to the observed result Rts corresponds, except that the impact metric RT5 continuously distributed between 0 and 100 and not a discrete distribution, such as... Rts The impact metrics 410 describe traffic scenarios using impact metrics. RT5 which are greater than a user-defined threshold, which may be, for example, 80, are recorded and stored in a memory and are marked by the system 400 for the investigation of simulated traffic scenarios as traffic scenarios 412 with a high probability of a collision or near-collision.

[0043] Fig. Figure 5 is a representation of System 500 for the utilization of simulated traffic scenarios. System 500 utilizes the traffic scenarios with a high probability of a collision or near-collision, generated by System 400 for the study of simulated traffic scenarios, by randomly perturbing (i.e., modifying) the initialization and setup data for the traffic scenarios identified as traffic scenarios 412 with a high probability of a near-collision or collision. For example, a random number generator can generate random numbers to be applied to parameters included in the initialization and setup data. The random numbers are conditioned to generate perturbations on the parameters included in the initialization and setup data, meaning that the changes to the parameters are small relative to their magnitude.Small changes, for example on the order of one or two percent in a parameter, generate a traffic scenario that is different from, but similar to, the traffic scenario generated from undisturbed initialization and setup data. In this way, if a traffic scenario has a high probability of involving a collision or near-collision, the traffic scenarios generated based on the disturbed data can be reviewed to determine whether the disturbed data also leads to a collision or near-collision.The System 500 for using simulated traffic scenarios generates multiple traffic scenarios for each traffic scenario 412 with a high probability of near-collision or collision, in order to provide multiple versions of the traffic scenarios 412 with a high probability of near-collision or collision, which include multiple variations of the traffic scenarios 412 with a high probability of near-collision or collision for training a machine learning system.

[0044] A jamming device 502 receives as input the traffic scenarios 412 with a high probability of a near-collision or collision. Fig. 4 together with the stored initialization and setup data for the traffic scenarios 412 with a high probability of near-collision or collision, which were stored during recording and saving 406. The jamming device 502 "rewinds" the traffic scenarios 412 by restarting the traffic scenario a user-defined number of seconds before a collision or near-collision event, for example, five seconds. The jamming device 502 then disrupts or modifies the parameters contained in the initialization and setup data by a small amount, starting from the user-defined point before the collision or near-collision event. For example, the starting locations and trajectories of the ego vehicle 210 and the second vehicle 214 can be changed. The environmental conditions can also be modified.The magnitude of the change can be randomly selected and conditioned to produce resulting traffic scenarios that exhibit similar probabilities of a collision or near-collision, but provide differently appearing video sequences and different ground-truth data to introduce variation into a training dataset. By providing variation in video sequences and ground-truth data while maintaining high probabilities of a collision or near-collision in the traffic scenarios, the ability of a machine learning system to detect collision or near-collision traffic scenarios in real-world traffic situations can be improved.

[0045] A rendering device 506 generates a traffic scenario based on disturbed initialization and setup data using photorealistic rendering software, as described above in relation to Fig. 3 discussed. After rendering the traffic scenario, the Record and Store 508 stores the corrupted initialization and setup data, along with the video sequence and ground truth data included in the traffic scenario generated by the Renderer 506, in the memory included in the System 500 for using simulated traffic scenarios in the Server Computer 120. Following the Record and Store 508, an Impact Evaluation 510 determines the probabilities of a collision or near-collision using a random forest regressor, as discussed above with respect to Fig. 4 described.

[0046] Following the determination of the probability of a collision or near-collision in the Collision Rating 510, the probability of a collision or near-collision is compared to a user-defined minimum in a Probability Comparison 512. The Probability Comparison 512 compares the probability of a collision or near-collision to a user-defined minimum, such as 80 percent, to ensure that the given traffic scenario includes a high probability of a near-collision or collision before the traffic scenario is included in a training dataset to train a machine learning system. The System 500 for Simulated Traffic Scenario Use continues to generate disrupted traffic scenarios until the user-defined minimum is reached.

[0047] After determining the probability of a collision or near-collision, the traffic scenario is added to the current set of perturbed traffic scenarios and passed to a Convergence Test 514. The Convergence Test compares the probability distributions included in the current set of traffic scenarios with probability distributions included in previous sets of traffic scenarios using Kullback-Leibler (KL) divergence. KL divergence measures the distance between probability distributions. The KL divergence measured between probability distributions from the current set of perturbed traffic scenarios and probability distributions from previous sets of perturbed traffic scenarios can be compared to a user-defined minimum distance a.The user-defined minimum distance a can be empirically determined by reviewing sets of perturbed traffic scenarios and determining when additional perturbed traffic scenarios would not noticeably change the measured KL divergence. A measured KL divergence less than a user-defined minimum distance a indicates that the current set of perturbed traffic scenarios generated by System 500 for Simulated Traffic Scenarios has converged to a stable solution, and the perturbed traffic scenario generation process can end. Once the perturbed traffic scenario generation process converges, the generated perturbed traffic scenarios can be stored in memory, and System 500 for Simulated Traffic Scenarios returns to utilizing the next traffic scenario 412 in the queue.Following the convergence test 514, traffic scenarios with a high probability of a collision or near-collision are transferred to a training dataset, which may be contained in a memory located in the server computer 120.

[0048] After the system 500 generates traffic scenarios with a high probability of a collision or near-collision for use in simulated traffic scenarios and transfers them to a training dataset, the training dataset can be used to train a machine learning system, for example, a convolutional neural network, to handle collision and near-collision traffic situations based on input from a vehicle 110 via sensor 116. For example, a machine learning system on the server computer 120 can be trained based on the training dataset and communicated to a computing device 115 in a vehicle 110. The trained machine learning system, running on the computing device 115, can receive sensor data from a sensor 116, for example, a video camera, which is included in a vehicle 110.In response to input from sensor 116, the machine learning system can output a prediction for vehicle 110, including the probability of a collision or near-collision with an object in the vicinity of vehicle 110. In response to a prediction indicating a high probability of a collision or near-collision with an object, the computing device 115 can determine a vehicle path that reduces the probability of a collision or near-collision with the object. A vehicle path is a predicted trajectory for the vehicle's movement, which includes lateral and longitudinal acceleration. The computing device 115 can then control one or more of the vehicle's propulsion, steering, and braking systems via controllers 112, 113, and 114 to cause vehicle 110 to travel along the vehicle path and reduce the probability of a collision or near-collision.

[0049] Fig. Figure 6 is a flowchart of a process 600 for generating traffic scenarios with a system 400 for investigating simulated traffic scenarios. Process 600 can, for example, be implemented on a server computer 120. Process 600 comprises several blocks that can be executed in the illustrated sequence. Alternatively or additionally, Process 600 could include fewer blocks and / or the blocks could be executed in a different sequence.

[0050] Process 600 begins at block 602, where initialization and setup data for generating traffic scenarios are determined. As previously mentioned regarding Fig. As described in Figures 3-5, the initialization and setup data can be determined based on user input or by processing real-world video sequences captured by sensors 116 contained in a vehicle 110 using a machine learning system, such as a convolutional neural network. The initialization and setup data includes initial locations, trajectories (including lateral and longitudinal accelerations for one or more objects, which may include vehicles, pedestrians, etc.), map data, and environmental conditions.

[0051] In block 604, the initialization and setup data are randomized to change the initial conditions, enabling process 600 to generate multiple traffic scenarios with different outcomes, including different probabilities of collision or near-collision between objects included in the traffic scenarios.

[0052] At block 606, the randomized initialization and setup data are received by the photorealistic rendering software to generate a traffic scenario that includes a video sequence and ground truth data, as described above in relation to Fig. 3-5 described above.

[0053] In block 608, the generated traffic scenario and the randomized initialization and setup data are recorded and stored on a memory that is included in the system 400 for the study of simulated traffic scenarios in a server computer 120.

[0054] At block 610, the probability of a collision or near-collision for the generated traffic scenario is determined using a random forest regressor, as described in relation to Fig. 4 described above.

[0055] At block 612, the probability of a collision or near-collision for the generated traffic scenario is compared to a user-defined threshold, as described above in relation to Fig. As described in section 4 above, if the probability of a collision or near-collision is less than the user-defined threshold, process 600 transitions to block 614. If the probability of a collision or near-collision is greater than or equal to the user-defined threshold, process 600 transitions to block 616.

[0056] In block 614, the number of completed traffic scenarios for process 600 is compared to a user-defined maximum number of traffic scenarios. If the number of completed traffic scenarios is less than the maximum number, process 600 increments the completed number of traffic scenarios and proceeds to block 604, where a new randomization is applied to the initialization and setup data, and a new traffic scenario is generated. If the number of completed traffic scenarios equals the user-defined maximum, process 600 terminates.

[0057] At block 616, it was determined that the probability of a collision or near-collision for the current traffic scenario is high. The traffic scenario is recorded and stored in memory within system 400, which is used to investigate simulated traffic scenarios on server computer 120, as a high-probability collision or near-collision traffic scenario 412. Following block 616, process 600 proceeds to block 614 to check if the maximum number of scenarios has been reached.

[0058] Fig. Figure 7 is a flowchart of Process 700 for generating disrupted traffic scenarios with a System 500 for using simulated traffic scenarios. Process 700 can, for example, be implemented on a server computer 120. Process 700 comprises several blocks that can be executed in the illustrated sequence. Alternatively or additionally, Process 700 could contain fewer blocks and / or the blocks could be executed in a different sequence.

[0059] Process 700 begins at block 702, where a seed traffic scenario is loaded from traffic scenarios 412 with a high probability of a collision or near-collision, which were generated by system 400 for investigating simulated traffic scenarios, as described above in relation to process 600. Fig. 6 described. The traffic scenario includes a video sequence, ground truth data, and initialization and setup data used to generate the traffic scenario.

[0060] In block 704, a random number generator produces random numbers that are used to perturb parameters included in the initialization and setup data. These random numbers are conditioned to generate perturbations in the parameters included in the initialization and setup data, as described above in relation to... Fig. 5 described.

[0061] At block 706, the disrupted initialization and setup data are received by the photorealistic rendering software to generate a traffic scenario that includes a video sequence and ground truth data, as described above in relation to Fig. 5 described above.

[0062] At block 708, the generated traffic scenario and the randomized initialization and setup data are recorded and stored on a memory that is included in the system 500 for the use of simulated traffic scenarios in a server computer 120.

[0063] At block 710, the probability of a collision or near-collision for the generated traffic scenario is determined using a random forest regressor, as described in relation to Fig. 4 described above.

[0064] In block 712, the number of completed traffic scenarios for process 700 is compared to a user-defined minimum number of traffic scenarios. If the number of completed traffic scenarios is less than the minimum number, process 700 increments the completed number of traffic scenarios and proceeds to block 704, where a new randomization is applied to the initialization and setup data, and a new traffic scenario is generated. If the number of completed traffic scenarios equals the user-defined minimum, process 700 proceeds to block 714.

[0065] At block 714, the distribution of probabilities included in the current disrupted traffic scenario is compared with distributions of probabilities included in previously generated traffic scenarios by determining the KL divergence between the distributions of probabilities for the current set of disrupted traffic scenarios and a previously determined set of disrupted traffic scenarios, comparing this to a user-defined minimum distance a, as described above in relation to Fig. 5. If the KL divergence is greater than the user-defined minimum distance a, process 700 has not converged and proceeds to block 704 to generate additional disruptive traffic scenarios based on the current initialization and setup data. If the KL divergence is less than the user-defined minimum distance a, process 700 terminates.

[0066] Fig. Figure 8 is a flowchart of a process 800 for operating a vehicle 110 with a machine learning system trained using a training dataset generated by a system 400 for investigating simulated traffic scenarios and a system 500 for utilizing simulated traffic scenarios. The process 800 can, for example, be implemented in a computing device 115 within a vehicle 110. The process 800 comprises several blocks that can be executed in the illustrated sequence. Alternatively or additionally, the process 800 could comprise fewer blocks and / or involve the blocks being executed in different sequences.

[0067] Process 800 begins at block 802, where a computing device 115 in a vehicle 110 downloads a machine learning system that was trained on a training database generated by a system 400 for investigating simulated traffic scenarios and a system 500 for using simulated traffic scenarios, as described in relation to Fig. 4 and Fig. 5 described above. The machine learning system can, for example, be a convolutional neural network that includes convolutional layers and fully connected layers.

[0068] In block 804, the computing device 115 acquires sensor data from one or more sensors 116. For example, the computing device 115 can input an image from a video sensor that contains image data relating to an environment around the vehicle 110.

[0069] At block 806, the machine learning system can receive the captured image and process it to determine the probability of a collision or near-collision with an object in the vicinity of vehicle 110.

[0070] At block 808, the computing device 115 can receive the probability of a collision or near-collision and determine a vehicle path that reduces the probability of a collision or near-collision. For example, the computing device 115 can determine a vehicle path that applies vehicle steering to steer the vehicle 110 away from the object or applies vehicle braking to slow the vehicle 110 down. The computing device 115 can transmit commands to the vehicle controls 112, 113, and 114 to control one or more of the vehicle's propulsion, steering, and braking systems to cause the vehicle 110 to travel along the determined vehicle path. Following block 808, process 800 terminates.

[0071] Any action taken by a vehicle or its user in response to one or more of the navigation prompts disclosed in this document should comply with all rules and regulations specific to the location and operation of the vehicle (e.g., federal, state, county, city, etc.). Furthermore, any navigation prompts disclosed in this document are for illustrative purposes only. Certain navigation prompts may be modified and / or omitted depending on the context, situation, and applicable rules and regulations. In addition, users should exercise good judgment and common sense when operating the vehicle, regardless of the navigation prompts.This means that all navigation prompts, whether standard or "enhanced", should be treated as suggestions and followed only if it is safe to do so and if the rules and regulations specific to the location and operation of the vehicle are observed.

[0072] Computing devices, such as those described in this document, generally each contain instructions that can be executed by one or more computing devices, such as those identified above, to carry out blocks or steps of processes described above. For example, process blocks described above may be implemented as computer-executable instructions.

[0073] Computer-executable instructions can be compiled or interpreted by computer programs created using a variety of programming languages ​​and / or technologies, including, without limitation, either alone or in combination, Java™, C, C++, Python, Julia, SCALA, Visual Basic, JavaScript, Perl, HTML, etc. Generally, a processor (e.g., a microprocessor) receives instructions, for example, from memory, a computer-readable medium, etc., and executes these instructions, thereby performing one or more processes that include one or more of the processes described in this document. Such instructions and other data can be stored in files and transmitted using a variety of computer-readable media. A file in a computing device is generally a collection of data stored on a computer-readable medium, such as a data storage medium, random-access memory, etc., is stored.

[0074] A computer-readable medium (also called a processor-readable medium) includes any non-transient (i.e., physical) medium involved in providing data (i.e., instructions) that can be read by a computer (i.e., by a computer's processor). Such a medium can take many forms, including but not limited to non-volatile and volatile media. Instructions can be transmitted through one or more transmission media, including optical fibers, wires, wireless communication, and internal components that comprise a system bus connected to a computer's processor. Common forms of computer-readable media include, for example, RAM, a PROM, an EPROM, a FLASH EEPROM, any other memory chip or cartridge, or any other medium from which a computer can read.

[0075] All terms used in the patent claims shall have their general and ordinary meaning as understood by a person skilled in the art, unless expressly stated otherwise in this document. In particular, the use of singular articles such as "a", "an", "the", "the", etc., shall be interpreted as referring to one or more of the listed elements, unless a patent claim expressly specifies otherwise.

[0076] The term “exemplary” is used in this writing to mean an example, i.e., a candidate for an “exemplary device” should simply be read as referring to an example of a device.

[0077] The adverb "approximately", which modifies a value or result, means that a form, structure, measurement, value, determination, calculation, etc., may deviate from a precisely described geometry, distance, measurement, value, determination, calculation, etc., due to deficiencies in materials, processing, manufacturing, sensor measurements, calculations, processing time, communication time, etc.

[0078] In the drawings, the same reference numerals denote the same elements. With regard to the media, processes, systems, procedures, etc., described in this document, it is understood that although the steps or blocks of such processes, etc., have been described as occurring in a certain sequence, such processes could be implemented in such a way that the described steps are carried out in a different order than that described in this document. Furthermore, it is understood that certain steps could be carried out simultaneously, other steps could be added, or certain steps described in this document could be omitted. In other words, the descriptions of processes in this document are provided to illustrate certain embodiments and should in no way be interpreted as limiting the claimed invention.

[0079] According to the present invention, a method comprises the following: generating first traffic scenarios, which include one or more objects, based on initialization and setup data for the one or more objects, wherein the first traffic scenarios are generated using a three-dimensional simulation engine; determining a probability of a collision between the one or more objects included in the first traffic scenarios; selecting a subset of the first traffic scenarios that includes a collision probability greater than a user-selected threshold; generating second traffic scenarios based on perturbing the selected subset of the first traffic scenarios, wherein the second traffic scenarios are generated using the three-dimensional simulation engine;and training a machine learning system based on the first and second traffic scenarios.

[0080] In one aspect of the invention, the method involves disrupting the selected subset of the first traffic scenarios by changing one or more parameters that are included in the initialization and setup data of one or more objects.

[0081] In one aspect of the invention, the three-dimensional simulation engine receives the initialization and setup data and determines these simulated video sequences, which include object motion paths for the one or more objects.

[0082] In one aspect of the invention, the simulated video sequences are based on photorealistic image rendering.

[0083] In one aspect of the invention, the object motion paths for the one or more objects are based on physical simulation.

[0084] In one aspect of the invention, the initialization and setup data includes initial locations and initial movement paths for the one or more objects.

[0085] In one aspect of the invention, the probability of an impact of one or more objects is based on four impact metrics and a random forest regressor.

[0086] In one aspect of the invention, the four impact metrics are based on longitudinal deceleration, maximum permissible longitudinal deceleration, longitudinal acceleration, maximum permissible longitudinal acceleration, lateral acceleration and maximum permissible lateral acceleration.

[0087] In one aspect of the invention, the machine learning system is a neural network that includes convolutional layers and fully connected layers.

[0088] In one aspect of the invention, the method involves transferring the trained machine learning system to a system that is included in a vehicle.

[0089] In one aspect of the invention, the method involves operating the vehicle based on the trained machine learning system.

[0090] In one aspect of the invention, the method involves operating the vehicle by controlling one or more of the vehicle's drive, steering, and braking systems.

[0091] According to the present invention, a system is provided comprising: a computer including a processor and memory, wherein the memory contains instructions that can be executed by the processor to: transmit a trained machine learning system, wherein the machine learning system has been trained with a training dataset determined by: generating initial traffic scenarios involving one or more objects based on initialization and setup data for the one or more objects, wherein the initial traffic scenarios are generated using a three-dimensional simulation engine; determining a probability of a collision between the one or more objects included in the initial traffic scenarios;Selecting a subset of the first traffic scenarios that includes a collision probability greater than a user-selected threshold; generating second traffic scenarios based on perturbing the selected subset of the first traffic scenarios, with the second traffic scenarios being generated using the three-dimensional simulation engine; and training the machine learning system based on the first and second traffic scenarios.

[0092] According to one embodiment, the invention is further characterized by the second computer for a vehicle, which includes a second processor and a second memory, wherein the second memory contains second instructions that can be executed by the second processor to receive and obtain an output from the trained machine learning system.

[0093] According to one embodiment, the invention is further characterized by additional instructions for determining a vehicle path based on the trained machine learning system.

[0094] According to one embodiment, the invention is further characterized by additional instructions for operating the vehicle based on the vehicle's path.

[0095] According to one embodiment, the invention is further characterized by additional instructions for operating the vehicle by controlling one or more of the vehicle drive, vehicle steering and vehicle brakes.

[0096] According to one embodiment, the invention is further characterized by additional instructions for disrupting the selected subset of the first traffic scenarios by changing one or more parameters that are included in the initialization and setup data of one or more objects.

[0097] According to one embodiment, the three-dimensional simulation engine receives the initialization and setup data and determines these simulated video sequences, which include object motion paths for the one or more objects.

[0098] According to one embodiment, the simulated video sequences are based on photorealistic image rendering.

Claims

[1] Method comprising: Generating first traffic scenarios including one or more objects based on initialization and setup data for the one or more objects, wherein the first traffic scenarios are generated using a three-dimensional simulation engine; Determining a probability of an impact between the one or more objects included in the first traffic scenarios; Selecting a subset of the first traffic scenarios that includes a probability of impact greater than a user-selected threshold; Generating second traffic scenarios based on perturbing the selected subset of the first traffic scenarios, wherein the second traffic scenarios are generated using the three-dimensional simulation engine; and Training a machine learning system based on the first and second traffic scenarios. [2] The method of claim 1, further comprising perturbing the selected subset of the first traffic scenarios by changing one or more parameters included in the initialization and setup data of the one or more objects. [3] The method of claim 1, wherein the three-dimensional simulation engine receives the initialization and setup data and determines simulated video sequences including object trajectories for the one or more objects. [4] The method of claim 3, wherein the simulated video sequences are based on photorealistic image rendering. [5] The method of claim 4, wherein the object trajectories for the one or more objects are based on physical simulation. [6] The method of claim 1, wherein the initialization and setup data includes initial locations and initial trajectories for the one or more objects. [7] The method of claim 1, wherein the probability of impact of the one or more objects is based on four impact metrics and a Random Forest regressor. [8] The method of claim 7, wherein the four impact metrics are based on a longitudinal deceleration, a maximum allowable longitudinal deceleration, a longitudinal acceleration, a maximum allowable longitudinal acceleration, a lateral acceleration, and a maximum allowable lateral acceleration. [9] The method of claim 1, wherein the machine learning system is a neural network including convolutional layers and fully connected layers. [10] The method of claim 1, further comprising transferring the trained machine learning system to a system included in a vehicle. [11] The method of claim 10, further comprising operating the vehicle based on the trained machine learning system. [12] The method of claim 11, further comprising operating the vehicle by controlling one or more of vehicle propulsion, vehicle steering, and vehicle braking. [13] The method of claim 7, wherein the random forest regressor includes multiple decision trees. [14] The method of claim 13, wherein the plurality of decision trees randomly sample the four impact metrics across multiple traffic scenarios. [15] A system comprising a computer programmed to perform the methods of any one of claims 1-14.

Images