Future trajectory prediction in multi-actor environments for autonomous machine applications

Abstract

In various examples, past location information and map information corresponding to actors in the environment can be applied to a deep neural network (DNN) trained to compute information corresponding to future trajectories of the actors, such as a recurrent neural network (RNN). The output of the DNN can include, for each future time slice that the DNN is trained to predict, a confidence map of the confidence of each pixel representing the presence of an actor and a vector field representing the location of the actor in the confidence map of the previous time slice. The vector field can thus be used to track the object through the confidence map of each future time slice to generate a predicted future trajectory for each actor. In addition to the tracked past trajectories, the predicted future trajectories can be used to generate a complete trajectory for an actor of the ego vehicle that can assist in navigating the environment.

Description

Background Art

[0001] For autonomous vehicles to navigate effectively, they need to develop an understanding of their surroundings. For example, identifying the location of nearby cars, pedestrians, traffic signs and signals, and road configurations are key aspects of safe control. Determining the likely future trajectories of objects over time, in addition to their current location and configuration within the environment, through observation by the autonomous vehicle, can prove effective in fully understanding and accounting for predicted changes in the environment.

[0002] Conventional systems use a single-actor approach to leverage past object trajectory information, where an understanding of the actors surrounding the ego vehicle can be determined, and the future trajectory of each actor can then be individually calculated. For example, some conventional systems rely on a combination of networks to determine high-level features corresponding to the actors in the environment, and then perform individual calculations for each actor based on the high-level features. These systems may or may not predict multi-path trajectories or single-mode trajectories. Other conventional systems may calculate possible future trajectories for actors in the environment, such as by using a variational autoencoder (VAE). However, these conventional systems do not calculate future trajectories based on the past trajectories of the corresponding actors, but rather calculate potential future trajectories based on historical information about possible trajectories given a certain map structure. Consequently, these conventional approaches are limited to predicting the future position of a single object at a time (thus limiting the usefulness of this information for autonomous driving applications) or predicting many possible trajectories that do not directly correspond to the actual predicted future trajectory of the actor. When these conventional processes are repeated for each object, the system's runtime becomes inefficient for real-time deployment due to the processing burden on the system. Summary of the Invention

[0003] Embodiments of the present disclosure relate to future trajectory prediction in a multi-actor environment for autonomous machine applications. Systems and methods are disclosed for predicting the future locations of agents (e.g., using a deep neural network (DNN), such as a recurrent neural network (RNN)) using previously tracked agent locations and map information. In addition to previous locations and map information, the system can also utilize wait conditions, free space boundary information, and / or other environmental information to compute accurate and reliable predictions of future agent locations.

[0004] In contrast to conventional systems (such as those described above), the present system simultaneously utilizes deep neural networks (DNNs), such as recurrent neural networks (RNNs), to compute the future trajectories of any number of agents. Information from the perception stack (e.g., as generated using one or more sensors of the vehicle) can be used to compute the previous positions of agents in the environment. This information, in addition to map information (e.g., the locations of static objects, waiting condition information, free space boundary locations, road structure information, etc.), can also be applied to the DNN. In some embodiments, to avoid accuracy issues associated with perspective overhead or overhead orthographic projection view data during processing by the DNN, the map information and previous position information can be mapped to an overhead view and rasterized to generate an image representing the overhead view. Using this information, the DNN can compute confidence maps (e.g., representing confidence levels corresponding to the presence of an agent) and vector fields for any number of future time slices, and the vector fields can be used to track the agent from time slice to time slice—e.g., in reverse chronological order. In some embodiments, clustering and / or weighted averaging can be used to determine the final future position of each agent, such as when multiple points in the confidence map indicate the presence of the agent. Thus, the ego vehicle can generate and use trajectories representing the past and / or future positions of each agent to help navigate the environment, e.g., for path planning, control decisions, obstacle avoidance, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The present system and method for future trajectory prediction in a multi-actor environment for autonomous machine applications is described in detail below with reference to the accompanying drawings, wherein:

[0006] Figure 1A An example data flow diagram including a process for predicting the trajectory of one or more agents in an environment according to some embodiments of the present disclosure;

[0007] Figure 1B Describes some embodiments of the present disclosure suitable for Figure 1A an example deep neural network (DNN) architecture implemented in at least one embodiment of the process of;

[0008] Figures 2A-2B depicts a visual representation of an example input to a DNN according to some embodiments of the present disclosure;

[0009] Figures 3A-3B depicts a visual representation of example output from a DNN according to some embodiments of the present disclosure;

[0010] Figure 3C depicts using example output from a DNN to generate a visual representation of a trajectory of an agent in an environment, according to some embodiments of the present disclosure;

[0011] Figure 4A depicts a visual representation of an example trajectory of an active person overlaid on a map according to some embodiments of the present disclosure;

[0012] Figure 4B depicts a visual representation of an agent, associated trajectories, waiting conditions, and road structure according to some embodiments of the present disclosure;

[0013] Figure 5 is a flowchart illustrating a method for computing actor trajectories using a DNN according to some embodiments of the present disclosure;

[0014] Figure 6A is an illustration of an example autonomous vehicle according to some embodiments of the present disclosure;

[0015] Figure 6B According to some embodiments of the present disclosure, Figure 6A Examples of camera positions and fields of view for autonomous vehicles;

[0016] Figure 6C According to some embodiments of the present disclosure, Figure 6A a block diagram of an example system architecture for an example autonomous vehicle;

[0017] Figure 6D According to some embodiments of the present disclosure Figure 6A A system diagram of communications between a cloud-based server and an example autonomous vehicle; and

[0018] Figure 7 is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0019] Systems and methods are disclosed relating to future trajectory prediction in a multi-actor environment for autonomous machine applications using one or more deep neural networks (DNNs). Although the present disclosure may be described with respect to an example autonomous vehicle 600 (alternatively referred to herein as "vehicle 600" or "ego-vehicle 600"), 6A to 6DExamples thereof are described, but this is not intended to be limiting. For example, the systems and methods described herein may be used by, but are not limited to, non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more adaptive driver assistance systems (ADAS)), robots, warehouse vehicles, off-road vehicles, flying watercraft, boats, shuttles, emergency response vehicles, motorcycles, electric bicycles or motorized bicycles, aircraft, construction vehicles, underwater vessels, drones, and / or other vehicle types. Furthermore, although the present disclosure may be described with respect to autonomous driving or ADAS systems, this is not intended to be limiting. For example, the systems and methods described herein may be used in simulated environments (e.g., to more accurately simulate the movement of objects and control thereof in a simulated environment), robots, aerial systems, boating systems, and / or other technical fields, such as for perception, world model management, path planning, obstacle avoidance, and / or other processes.

[0020] A system using DNN for future trajectory prediction

[0021] refer to Figure 1A , Figure 1A is an example data flow diagram of a process 100 for predicting the trajectory of one or more agents in an environment according to some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are set forth only as examples. In addition to or in lieu of Figure 1A In addition to the arrangements and elements shown in the drawings, other arrangements and elements (e.g., machines, interfaces, functions, orders, functional groupings, etc.) may be used, and some elements may be omitted entirely. Further, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in combination with other components, and in any suitable combination and position. Each function described herein as being performed by an entity may be performed by hardware, firmware, and / or software. For example, each function may be implemented by a processor executing instructions stored in a memory.

[0022] Process 100 may include generating and / or receiving sensor data 102 from one or more sensors of vehicle 600. Sensor data 102 may be used by vehicle 600, and within process 100, to predict future trajectories of one or more objects or agents in the environment (such as other vehicles, pedestrians, cyclists, etc.). Sensor data 102 may include, but is not limited to, sensor data 102 from any sensor of vehicle 600 (and / or other vehicles or objects, such as robotic devices, VR systems, AR systems, etc., in some examples). For example, and with reference to Figures 6A to 6C, the sensor data 102 may include data generated by, but is not limited to, a global navigation satellite system (GNSS) sensor 658 (e.g., a global positioning system sensor), one or more RADAR sensors 660, one or more ultrasonic sensors 662, one or more LIDAR sensors 664, one or more inertial measurement unit (IMU) sensors 666 (e.g., one or more accelerometers, one or more gyroscopes, one or more magnetic compasses, one or more magnetometers, etc.), one or more microphones 696, one or more stereo cameras 668, one or more wide-angle cameras 670 (e.g., fisheye cameras), one or more infrared cameras 672, one or more surround cameras 674 (e.g., 360-degree cameras), one or more long-range and / or mid-range cameras 698, a speed sensor 644 (e.g., for measuring the speed and / or distance traveled of the vehicle 600), and / or other sensor types.

[0023] In some examples, sensor data 102 may include sensor data generated by one or more forward-facing sensors, side-view sensors, and / or rear-view sensors. This sensor data 102 may be used to identify, detect, classify, and / or track the movement of objects around vehicle 600 within the environment. In embodiments, any number of sensors may be used to combine multiple fields of view (e.g., Figure 6B The remote camera 698, the forward stereo camera 668 and / or the forward wide-angle camera 670) and / or the sensing field (e.g., the sensing field of the LIDAR sensor 664, the RADAR sensor 660, etc.).

[0024] The sensor data 102 may include image data representing an image, image data representing a video (e.g., a snapshot of a video), and / or sensor data representing a representation of a sensor's field of view (e.g., a depth map of a LIDAR sensor, a value map of an ultrasonic sensor, etc.). Where the sensor data 102 includes image data, any type of image data format may be used, such as, but not limited to, compressed images such as Joint Photographic Experts Group (JPEG) or Luma / Chroma (YUV) formats, compressed images derived from frames of a compressed video format such as H.264 / Advanced Video Coding (AVC) or H.265 / High Efficiency Video Coding (HEVC), raw images such as from red-transparent-blue (RCCB), red-transparent-blue (RCCC), or other types of imaging sensors, and / or other formats. Furthermore, in some examples, sensor data 102 may be used within process 100 without any preprocessing (e.g., in a raw or captured format), while in other examples, sensor data 102 may undergo preprocessing (e.g., noise balancing, demosaicing, scaling, cropping, enhancement, white balancing, tone curve adjustment, etc., such as using a sensor data preprocessor (not shown). As used herein, sensor data 102 may refer to unprocessed sensor data, preprocessed sensor data, or a combination thereof.

[0025] Additionally, the process 100 may include accessing and / or storing a map from the vehicle 600, such as the HD map 104 (which may be similar to Figure 6C HD map 104)—generates and / or receives map data. In some embodiments, HD map 104 may include accuracy down to the finest centimeter level, allowing vehicle 600 to rely on HD map 104 for precise guidance, planning, and positioning. HD map 104 may represent lanes, road boundaries, road shape, elevation, slope and / or contours, heading information, waiting conditions, static object locations, and / or other information. As such, process 100 may use information from HD map 104 (such as the location and shape of corridors) to generate input 108 for DNN 116.

[0026] In addition to or in lieu of sensor data 102 and / or HD map 104, process 100 may also include generating and / or receiving (e.g., in embodiments, using sensor data 102 and / or HD map 104) one or more outputs from an autonomous or semi-autonomous (e.g., ADAS) driving software stack. For example, information generated by the perception layer, world model management layer, control layer, actuation layer, obstacle avoidance layer, and / or other layers of the software stack may be used within process 100 to generate input 108. This information may include free space boundary locations, waiting conditions, intersection structure detection, lane type identification, road shape information, object detection and / or classification information, and / or the like. As such, sensor data 102, HD map 104, and / or other information generated by vehicle 600 may be used to generate input 108 for DNN 116.

[0027] In some non-limiting embodiments, the sensor data 102, information from the HD map 104, and / or other information (e.g., from the driving stack) may be applied to a perspective shifter 106 before being used as an input 108 to the DNN 116. The perspective shifter 106 may orient the data relative to one of the agents in the environment, relative to a location on the road surface, and / or relative to another feature represented by the data. For example, in some embodiments, the perspective shifter 106 may shift the perspective of the data relative to the location and / or orientation of the vehicle 600 (e.g., the ego vehicle or the ego agent). Thus, the location of the agent or object, a portion of the HD map 104, and / or other information used as input 108 may be shifted relative to the vehicle 600 (e.g., with the ego vehicle 600 centered at (x, y) coordinates (0, 0), where y is the longitudinal dimension extending from the front to the rear of the vehicle and x is the lateral dimension perpendicular to y and extending from the left to the right of the vehicle, similar to Figure 4B). In some embodiments, in addition to or as an alternative to shifting the perspective relative to features of the environment, the perspective shifter 106 may shift the perspective to the same field of view. For example, where the HD map 104 may generate data from a top-down perspective of the environment, the sensor generating the sensor data 102 may do so from a different perspective (such as forward, sideways, angled downward, angled upward, etc.). As such, to produce inputs 108 that share the same perspective, the perspective shifter 106 may adjust each of the inputs 108 to the same perspective. In some non-limiting embodiments, each of the sensor data 102, the HD map 104, and / or other information may be shifted to a top-down perspective—e.g., a perspective top-down perspective and / or an orthographic top-down perspective. Furthermore, the perspective shifter 106 may facilitate generating the inputs 108 such that the same or substantially similar (e.g., to within centimeters, meters, etc.) portion of the environment is represented from that perspective for each instance of the inputs 108. For example, a first input (e.g., a rasterized image) representing a past position 110 of an active person in an environment may be represented by a top-down perspective of that portion of the environment, and a second input (e.g., a rasterized image) representing map information 112 of the environment may be depicted by a top-down perspective of that portion of the environment. Thus, the DNN 116 may use any number of inputs 108 corresponding to the same general portion of the environment and, therefore, generate outputs 118 at the same scale. However, this is not intended to be limiting, and in some embodiments, the perspective, orientation, size, location, and scale of the inputs 108 may be different for different input types and / or instances.

[0028] Input 108 may include past locations 110 (e.g., of an agent in the environment (e.g., a vehicle, pedestrian, cyclist, robot, drone, watercraft, etc.), state information 132 (e.g., velocity and / or acceleration data corresponding to the agent), map information 112 (e.g., as generated using HD map 104), wait conditions 114 (e.g., generated using sensor data 102, HD map 104, and / or other information), and / or other input 108 (e.g., free space information, static object information, etc., as determined using sensor data 102, HD map 104, the vehicle 600's driving stack 128, and / or other information). Past locations 110 may include previously detected locations of vehicles, pedestrians, cyclists, and / or other agent types in the environment. In some embodiments, past locations 110 may be determined relative to the ego vehicle 600 so that changes in orientation and position relative to the agent during perspective shifting are more efficiently implemented. Past locations 110 and / or state information 132 may be represented by an image (e.g., a rasterized image) representing the location of the activity. In some embodiments, each instance of past locations 110 may include a single image and may correspond to a single time slice, e.g., an instance may capture each activity being tracked and / or detected and their current location (e.g., relative to vehicle 600) at that time slice. In some embodiments, each instance of state information 132 may include a single image and may correspond to a single time slice. In other embodiments, state information 132 may be included in an image instance along with past locations 110. DNN 116 may take one or more instances of past locations 110 and / or state information 132 as input, such that DNN 116 may use one or more instances of past locations 110 and / or state information 132 corresponding to the activity's location over one or more time slices (e.g., within a time period) to compute output 118.

[0029] For example, relative to Figure 2A , each input 108 corresponding to the time slice at time T1 may include the past position 110A (and / or may include the state information 132 corresponding thereto). Thus, in an embodiment, each black oval may correspond to the position and / or state information of an agent in the environment, including the ego-vehicle 600. Similarly, with respect to Figure 2B , for the time slice at time T2, actors 210A-210G may be detected at various locations within the environment. Figure 2B The visualization 202 may represent the past locations 110 and the map information 112 at time T2. As a non-limiting example, the respective locations of each of the actors 210 may be oriented relative to the self-actor (which may be the centrally located actor 210E in the visualization 202) such that the DNN 116 may depend on the self-actor.

[0030] Map information 112 may include the locations of lanes (e.g., lane centerlines or tracks, lane edges or dividers, road boundaries, emergency lanes, etc.), the locations of static objects, the locations of intersections, road shape information, and / or the like. In some embodiments, map information 112 may be determined relative to ego vehicle 600 so that changes in orientation and position relative to the map information during viewpoint shifts are more efficiently achieved. Map information 112 may be represented by an image (e.g., a rasterized image) representing lane locations, static object locations, etc. In some embodiments, each instance of map information 112 may include a single image and may correspond to a single time slice, e.g., an instance may capture driving surface structure (e.g., relative to vehicle 600) in that time slice. DNN 116 may use one or more instances of map information 112 as input, such that DNN 116 may use one or more instances of map information 112 corresponding to road structure information at respective time slices (e.g., within a time period) to compute output 118. In some non-limiting embodiments, the same map information 112 may be used for each time slice within a time period (e.g., the same instance of map information 112 may be used for every two time slices, every three time slices, etc., and then updated at the same intervals). In other embodiments, the map information 112 may be updated at every time slice.

[0031] For example, relative to Figure 2A , each input 108 corresponding to the time slice at time T1 may include map information 112A. Thus, the map information 112A may include lane lines, line types, road shape and / or structure, and / or other features. Similarly, with respect to Figure 2B , for the time slice at time T2, may represent a road structure. As a non-limiting example, the map information 112 may be oriented relative to the self-actor (which may be the centrally located actor 210E in the visualization 202) such that the DNN 116 may depend on the self-actor.

[0032] The wait conditions 114 may include the location of a stop light, a yield sign, a stop sign, a building, an intersection (or the location of an intersection subject to such a location), and / or other wait conditions. In some embodiments, the wait conditions 114 may be included in the map information 112, while in other embodiments, the wait conditions 114 may represent a separate input channel to the DNN 116. In some embodiments, similar to the past positions 110 and / or the map information 112, the wait conditions 114 may be determined relative to the ego vehicle 600 so that changes in orientation and position relative to the wait conditions 114 during viewpoint shifts are more efficiently achieved. The wait conditions 114 may be represented by an image (e.g., a rasterized image) representing the location and / or type of the wait condition in the environment. In some embodiments, each instance of the wait condition 114 may include a single image and may correspond to a single time slice, e.g., the instance may capture the wait condition (e.g., relative to the vehicle 600) at that time slice. The DNN 116 may take as input one or more instances of the wait condition 114, such that the DNN 116 may use the one or more instances of the wait condition corresponding to the location and / or type of the wait condition at each time slice (e.g., within a time period) to compute the output 118. In some non-limiting embodiments, the same wait condition 114 may be used for each time slice within a time period (e.g., the same instance of the wait condition 114 may be used for every two time slices, every three time slices, etc., and then may be updated at the same interval). In other embodiments, the wait condition 114 may be updated at every time slice. For example, with respect to Figure 2A , each input 108 corresponding to the time slice at time T1 may include a wait condition 114A. As such, the wait condition 114A may include a stop sign, a stop light, a yield sign, an emergency vehicle entering a position, and / or other wait condition types.

[0033] The input 108 (e.g., after perspective shifting and / or rasterization) can be applied to the DNN 116 as an input tensor. For example, each respective input (e.g., map information 112, past positions 110, wait conditions 114, other input types, etc.) can each be applied as a separate input tensor to a channel of the DNN 116. As described herein, in some embodiments, each input type can be associated with an individual input tensor and / or input channel. In other embodiments, two or more input types (e.g., wait conditions 114 and map information 112) can be combined to form a single input tensor for a single input channel of the DNN 116.

[0034] In some embodiments, the DNN 116 may include a temporal and / or spatial DNN, such that the DNN 116 analyzes information corresponding to more than one time slice (e.g., time period) at each instance and / or analyzes information corresponding to more than one spatial location of the agent at each instance. As such, the DNN 116 may learn to predict future trajectories—or information representative thereof—by monitoring and considering the agent's past locations, road structures, waiting conditions, and / or other information over multiple time slices. In some embodiments, the DNN 116 may include a recurrent neural network (RNN). As a non-limiting example, and as described below, see Figure 1B Described in more detail, DNN 116 may include an encoder-decoder RNN 116A.

[0035] While examples are described herein with respect to using neural networks (specifically, RNNs) as DNNs 116, this is not intended to be limiting. For example, and without limitation, the DNNs 116 described herein may include any type of machine learning model, such as machine learning models using linear regression, logistic regression, decision trees, support vector machines (SVMs), naive Bayes, k-nearest neighbors (KNNs), K-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoders, convolutions, recurrent, perceptrons, long / short term memory (LSTM), Hopfield, Boltzmann, deep belief, deconvolution, generative adversarial, liquid state machines, etc.), and / or other types of machine learning models.

[0036] Now see Figure 1B , Figure 1B An example architecture of a DNN 116 is depicted. The DNN 116A includes a plurality of encoder-decoder stacks, each of which may include a 2D convolutional encoder 142 (e.g., 142A-142D), a 2D convolutional decoder 144 (e.g., 144A-144D), and / or a 2D convolutional RNN 146 (e.g., 146A-146D). The CNN 116A may be configured to receive any number of time slices worth of past information and predict any number of time slices worth of future information, depending on the embodiment. For example, the CNN 116A may generate a trajectory that includes information over the past two seconds and two seconds into the future, e.g., output trajectory points every second, every half second, four times a second, eight times a second, etc. The inputs 108A-108D may be similar to the output trajectory points with respect to Figures 2A-2B The input 108 described, and the outputs 118A-118D may be similar to those with respect to Figures 3A-3Coutputs described. For example, the inputs 108 may include tensors corresponding to past and / or predicted future locations of the agent, tensors corresponding to wait conditions 114, tensors corresponding to map information 112, and the like. The outputs 118 may include tensors corresponding to confidence fields, tensors corresponding to vector fields, and the like. In some embodiments, because the inputs 108 in the closed-loop mode are based on the actual (e.g., ground truth) location of the agent in the environment, the outputs 118 in the closed-loop mode may be more precise—e.g., may include a smaller area of ​​the agent's potential location, which may more closely approximate a 1:1 correspondence between the inputs 108 and the outputs 118. Furthermore, because the inputs 108 in the open-loop mode may be based on future predictions of the agent's location, the outputs 118 in the open-loop mode may be less precise—e.g., may include a larger area of ​​the agent's potential location, as described herein at least with respect to Figures 3A-3C described.

[0037] CNN 116A may include a past closed-loop model and a future open-loop model. In some embodiments, the past closed-loop model may take as input 108 the actual past position 110 of an agent in the environment (in addition to other inputs 108, such as map information 112, wait conditions 114, etc.) to generate output 118—e.g., as indicated by the square boxes on inputs 108A and 108B. The future open-loop model may take as input 108 the predictions of 2D convolutional decoder 144B based on the actual past position 110 of the agent as predicted by DNN 116A (e.g., as indicated by the black-filled circle and arrow 150A), and / or may take as input a future prediction of the agent's position as predicted by DNN 116A (e.g., as indicated by the white-filled circle and arrow 150B). As such, the output 108 in closed-loop mode can be based on the actual tracked position of the agent in the environment, and in open-loop mode can be based on the actual tracked position of the agent and / or the future predicted position of the agent. For example, state can be shared between the various encoder-decoder stacks 140, as indicated by the arrows between 140A and 140B, between 140B and 140C, and so on. State information can be passed so that predictions by encoder-decoder stack 140B take state information from encoder-decoder stack 140A into account, and so on.

[0038] See again Figure 1A, the output 118 of the DNN 116 may include one or more confidence fields 120, one or more vector fields 122, and / or other output types. The combination of the confidence field 120 and the vector field 122 may be used by a post-processor 124 (described in more detail herein) to determine a complete trajectory of the agent in the environment, which may include one or more past trajectory points or positions and / or one or more future trajectory points or positions. In some non-limiting embodiments, the confidence field 120 and the vector field 122 for a time slice may correspond to the same area of ​​the environment (e.g., the same region) and, therefore, may have the same spatial dimensions.

[0039] For each time slice (e.g., past, present, and / or future), the confidence field 120 may include a confidence field or map that represents the confidence of where the actor is located. The confidence field 120 may be represented by an H x W matrix, where each element (e.g., pixel or point) represents a confidence score. For example, each pixel or point in the confidence field 120 or map may have an associated confidence that the actor is present. Thus, and particularly for future predictions, one or more confidence fields 120 may look more like Figure 3A For example, Figure 3A The visualization 302 may represent multiple confidence fields 120 corresponding to multiple time slices that overlap with each other. Figure 3C A separate time slice is included for each confidence field 120A-120C, but visualization 302 may include multiple time slices compressed into a single plane corresponding to an area in the environment (e.g., from a top-down perspective). This visualization 302 can be helpful for visualizing static and dynamic or moving objects over time. For example, regions 304A and 304D may correspond to static actors and, therefore, may be represented by a substantially circular shape, while regions 304B and 304C may correspond to moving or dynamic actors and, therefore, may be represented by an elliptical shape, or otherwise indicate multiple predictions corresponding to different locations of the actors in the environment over time.

[0040] For each time slice (e.g., past, present, and / or future), the vector field 122 may include a vector field 122 or map representing a predicted vector (e.g., a displacement vector) corresponding to the location of the agent at the vector location at the previous time slice. The vector field 122 may include an H x W matrix in which each element (e.g., a pixel or point) represents a 2D (or 3D, in embodiments) vector corresponding to the displacement from the current vector location to a point (e.g., a center point) of the same object or agent in the previous time slice (or time step). In some non-limiting embodiments, each vector may be represented by a direction and magnitude, a distance along a 2D or 3D space (e.g., a pixel distance), and / or another representation. For example, at time Tn Each pixel or point in the vector field 122 or map may have an associated vector that represents the activity of the actor (if the actor were present at the pixel or point) at the previous time T n-1 is predicted to be located where (although in various embodiments, the DNN 116 may be trained to compute n+1 The corresponding vector field 122, for example). Figure 3B , the visualization 306 may represent multiple vector fields 122 overlapping each other and may similarly correspond to Figure 3A 3. The environment over the same time period as the visualization 302. The visualization 306 can further facilitate visualization of static and dynamic or moving objects over time. For example, regions 304A and 304D can correspond to static actors and, therefore, can be represented by substantially circular shapes, while regions 304B and 304C can correspond to moving or dynamic actors and, therefore, can be represented by elliptical shapes, or otherwise indicate multiple predictions corresponding to different locations of actors in the environment over time.

[0041] The post-processor 124 may use the confidence field 120 and the vector field 122 to determine the trajectory of each agent in the environment. For example, the last future time slice (e.g., T corresponding to the output 118) n ) can be analyzed by a post-processor 124 to determine the location of the active person, and the corresponding vectors from the vector field 122 of the same time slice can be used to determine the location of the active person from the previous time slice (e.g., T n-1 ) in the confidence field 120 of the previous time slice. The confidence field 120 from the previous time slice can then be used to determine the predicted location of the actor in the time slice (e.g., T n-1 ), and then the vector field 122 from that time slice can be used to determine the location of the actor from the previous time slice (e.g., T n-2 ), and so on, until the current time is reached. The trajectory generator 126 can then append these future predictions to the past trajectory of the actor determined from the actual detection of the actor to produce a final trajectory. In some embodiments, past trajectories can also be generated using a similar process as future trajectories, where the confidence field 120 is used to determine the position at the time slice, and the vector field 122 is used to determine the position at the previous time slice.

[0042] For a confidence field 120 corresponding to a time slice (e.g., as indicated by a timestamp), the location of the actor can be determined using any number of different methods, such as, but not limited to, a process involving clustering (e.g., non-maximum suppression, density-based spatial clustering with noise (DBSCAN), etc.) and / or a process without clustering. For example, where clustering is used, a confidence threshold can be applied to remove noise points. In such an example, the confidence threshold can be, but is not limited to, 0.7, 0.8, 0.85, 0.9, etc. Once the noise points are filtered out, the remaining points can have a clustering algorithm applied to them so that points within a threshold distance of each other can be determined to be associated with a single actor. In some embodiments, once a cluster is determined, one or more vectors corresponding to the same point from the vector field 122 of the same time slice can be used to find the location of the corresponding actor (or a cluster representing it) in the previous time slice. In other embodiments, once the clusters are determined, the centroid of each cluster can be determined, and a bounding shape of a predetermined size (e.g., the same size for all clusters, with different sizes for clusters of different agent types—e.g., a first-sized bounding shape for cars, a second-sized bounding shape for pedestrians, etc.) can be centered around the centroid (e.g., the centroid of the bounding shape centered around the centroid of the cluster). The bounding shape can then be used as a mask for the vector field 122 of the same time slice to determine which vectors to use to find the location of the corresponding agent (or a cluster or bounding shape representing it) in the previous time slice. This process can be repeated for each time slice until a complete trajectory through each time slice is determined. In examples where another agent (or a cluster or bounding shape representing it) is not located in the previous time slice using the vector field 122, the trajectory can be shortened, discarded (e.g., due to noise, defects, etc.), and / or estimated based on past temporal information.

[0043] As another example, without using clustering, another algorithm or method can be implemented to determine the location of the actors. For example, a weighted average method can be used where one or more confidence fields 120 and one or more vector fields 122 can be processed for each actor in a single pass - with the inherent computational benefit of fast processing times regardless of the number of actors. In such an algorithm, for each actor a, the most likely next position can be the average of all positions where its predecessor vector points to a, weighted by the confidence field 120 values ​​at these positions. The weighted average can be calculated once for all actors using auxiliary numerators and denominators stored - both initialized to zero. For each position pos in the output of the DNN 116, the predecessor, pred = predecessor [pos] and occupancy o = occupancy [pos]. Then o * pos is added to the numerator [pred], and o is added to the denominator [pred]. The next position a of each actor can be determined by numerator [a.position] / denominator [a.position]. The numerator stores the weighted sum of all positions whose predecessor vectors point to a, and the denominator stores the sum of their weights, so the result is a weighted average. Because the operations applied to each position are largely independent, these steps can be performed in parallel (for example, across multiple threads using a graphics processing unit (GPU) in parallel).

[0044] As another example, for each agent a, the confidence field 120 for a given time slice can be filtered to include pixels or points whose predecessor vectors point to agent a. A (soft) argmax function can be applied to the remaining points to determine the "center of mass" of these points. Specifically, the result can be the occupancy-weighted sum of all positions whose predecessors point to a. This can be determined as the most likely future position of a. The process can be repeated for each other agent. In some embodiments, a separate pass can be performed on the same confidence field 120 for each agent, and this can be repeated at each time slice. Therefore, the overall runtime of the system may be larger than required for real-time or near real-time deployment. To avoid this, and to perform per-agent operations on all agents jointly, two partial sums can be stored. According to the following equation (1), the first weighted sum of shape Hx W:

[0045]

[0046] and a second weighted sum of shape H x W x 2 according to the following equation (2):

[0047]

[0048] Then, to find the most likely successor of an actor, we can use equation (3)

[0049] sum_weighted_coords[a.bbox.sum() / sum_weights[a.bbox].sum() (3)

[0050] It may represent the occupancy-weighted average of all next-frame positions (or bounding boxes corresponding thereto) whose predecessors point to actor a.

[0051] In some instances, because occupancy scores (e.g., from confidence field 120) are not probabilities, a sharpening operation may be performed to avoid over-extending the trajectory. For example, a sharpening operation may be applied to confidence field 120 to assign higher weights to higher confidence score points before calculating the weighted average. In a non-limiting embodiment, the sharpening may be hard-coded with a sharpening strength 40, as represented in equation (4) below:

[0052] sharpen(x)=e 40·x-40 (4)

[0053] However, in some embodiments, the sharpening function may also be learned or trained.

[0054] As an example, relative to Figure 3C , the first time slice T N A plurality of different point groups 320 (e.g., 320A) may be included. Point groups (or clusters) 320 may be identified using clustering, weighted averaging, and / or other techniques (e.g., those described herein). For example, at time slice T N , point group 320A-1 may be determined (other point groups 320 may also be determined, but may be represented by other time slices (e.g., T N-1 and T N-2 ) occlusion), and can therefore determine the time slice T N The vector set 322A-1 of the corresponding vector field 122 (eg, the vector set from the vector field 122 corresponding to the same (x, y) coordinates as the point set 320 in the confidence field 120) may point to the time slice T. N-1 As a result, a connection can be established between the peer groups 320A-1 and 320A-2, belonging to the same actor, and used for the time slice T N and T N-1 Generate trajectory points. Similarly, in time slice T N-1 , it is possible to determine the point group 320A-2, and thus determine the points from the time slice T N-1 The corresponding vector set 322A-2 of the vector field 122. The vector set 322A-2 may point to the time slice TN-2 As a result, a connection can be made between peer groups 320A-2 and 320A-3, belonging to the same actor, and used for the time slice T N-2 Additional trajectory points are generated. Although not shown, this process can be repeated for any number of actors that appear in each time slice (e.g., including actors represented by point groups 320B-2, 320B-3, point groups 320C-2, 320C-3, and / or point group 320D-3). Furthermore, this process is not limited to three time slices and can be performed over any number of time slices depending on the embodiment. For non-limiting example, where one second in the past and one second in the future are to be included in the trajectory and the interval of calculation is six times per second, there can be twelve time slices.

[0055] Reference again Figure 1A , the trajectory generator 126 can use the output of the post-processor 124 to generate a trajectory for each active person being tracked. As an example, and referring to Figure 4A , Figure 4A Depicted is a visual representation of example trajectories of an agent overlaid on a map, according to some embodiments of the present disclosure. Visualization 400 may include map information, as indicated by lane lines 402, and trajectories 404A-404G for various agents (e.g., trajectory 404A for agent A, trajectory 404B for agent B, etc.). The black dots of trajectory 404 may indicate future position predictions, while the white dots may indicate past positions (e.g., known positions). In this manner, trajectory generator 126 may stitch together known past positions and predicted future positions and generate trajectory 404, which may be used by driving stack 128 of vehicle 600.

[0056] As another example, and with respect to Figure 4B , Figure 4BDepicted is a visual representation of agents, associated trajectories, waiting conditions, and road structure, according to some embodiments of the present disclosure. Visualization 410 may represent information passed to the driving stack 128 of vehicle 600 after process 100 has been executed. For example, visualization 410 may include an abstracted representation of a combination of inputs and outputs of DNN 116 (e.g., after post-processing). For example, road structure or map information from HD map 104 may be used to determine road boundaries 418, waiting conditions 114 may be used to determine the presence and locations of stop signs 416A-416D, respectively, and trajectories 414A-414F for each agent 412A-412F, respectively, may be determined based on the output of post-processor 124. Furthermore, as described herein, the representation may be egocentric, such that visualization 410 is centered from the perspective of the ego vehicle (e.g., agent 412C). The dashed lines of trajectory 414 may represent past known or tracked locations of agent 412, while the solid lines may represent predicted future locations of agent 412. The position of the actor 412 in the representation may represent the actor's location at the current time.

[0057] Reference again Figure 1A , the output of trajectory generator 126 can be transmitted or applied to driving stack 128 of vehicle 600. For example, once the trajectory has been calculated (and converted from 2D image space coordinates to 3D world space coordinates, in an embodiment), the trajectory can be used by autonomous vehicle 600 to perform one or more operations (e.g., obstacle avoidance, lane keeping, lane changing, path planning, mapping, etc.). More specifically, the trajectory can be used by driving stack 128 of autonomous vehicle 600, such as an autonomous machine software stack executed on one or more components of vehicle 600 (e.g., one or more SoCs 604, one or more CPUs 618, one or more GPUs 620, etc.). For example, vehicle 600 can use this information (e.g., the future location of one or more agents in the environment) to navigate, plan, or otherwise perform one or more operations (e.g., obstacle avoidance, lane keeping, lane changing, path planning, merging, splitting, etc.) within the environment.

[0058] In some embodiments, the trajectory can be used by one or more layers of the autonomous machine software stack 128 (alternatively referred to herein as the "driving stack 128"). The driving stack 128 can include a sensor manager (not shown), a perception component (e.g., corresponding to the perception layer of the driving stack 128), a world model manager, one or more planning components (e.g., corresponding to the planning layer of the driving stack 128), one or more control components (e.g., corresponding to the control layer of the driving stack 128), one or more obstacle avoidance components (e.g., corresponding to the obstacle avoidance layer or collision avoidance layer of the driving stack 128), one or more actuation components (e.g., corresponding to the actuation layer of the driving stack 128), and / or other components corresponding to additional and / or alternative layers of the driving stack 128. In some examples, process 100 can be performed by one or more perception components, which can feed output from one or more layers of the driving stack 128 to the world model manager, as described in more detail herein.

[0059] The sensor manager can manage and / or extract sensor data 102 from sensors of the vehicle 600. For example, and with reference to Figure 6C , sensor data 102 may be generated (e.g., permanently, at certain intervals, based on certain conditions) by one or more RADAR sensors 660. The sensor manager may receive sensor data 102 in different formats from the sensors (e.g., sensors of the same type may output sensor data in different formats) and may be configured to convert the different formats into a unified format (e.g., for each sensor of the same type). As a result, other components, features, and / or functions of the autonomous vehicle 600 may use the unified format, thereby simplifying the processing of the sensor data 102. In some examples, the sensor manager may use the unified format to apply controls back to the sensors of the vehicle 600, such as to set a frame rate or perform gain control. The sensor manager may also update sensor packets or communications corresponding to the sensor data with a timestamp to help inform the processing of the sensor data by various components, features, and functions of the autonomous vehicle control system.

[0060] The world model manager can be used to generate, update and / or define a world model. The world model manager can use information generated by one or more perception components of the driving stack 128 and received from one or more perception components of the driving stack 128 (e.g., past and predicted locations of detected active persons). One or more perception components may include obstacle sensors, path sensors, waiting sensors, map sensors, and / or other perception components. For example, the world model can be defined at least in part based on the affordances of obstacles, paths, and waiting conditions that can be perceived in real time or near real time by obstacle sensors, path sensors, waiting sensors, and / or map sensors. The world model manager can continuously update the world model based on the latest input (e.g., data) generated and / or received from obstacle sensors, path sensors, waiting sensors, map sensors, and / or other components of the autonomous vehicle control system.

[0061] The world model may be used to help inform one or more planning components, one or more control components, one or more obstacle avoidance components, and / or one or more actuation components of the driving stack 128. The obstacle sensor may perform obstacle perception, which may be based on where the vehicle 600 is permitted or able to drive (e.g., based on the location of a drivable path defined by avoiding detected obstacles), and how fast the vehicle 600 may drive without colliding with obstacles (e.g., objects such as structures, solids, vehicles, etc.) sensed by the vehicle's 600 sensors.

[0062] Path perception can perform path perception, such as by sensing the nominal path available in a particular situation. In some examples, path perception can further consider lane changes for path perception. A lane map (e.g., generated at least in part using HD map 104) can represent one or more paths available to vehicle 600 and can be as simple as a single path on an on-ramp on a highway. In some examples, the lane map can include a path to a desired lane and / or can indicate available changes along a highway (or other road type), or can include adjacent lanes, lane changes, forks, turns, intersections, merges, and / or other information.

[0063] As a result of rules, conventions, and / or practical considerations, the waiting sensor may be responsible for determining constraints on the vehicle 600. For example, the rules, conventions, and / or practical considerations may relate to traffic lights, multiple stops, yields, merges, toll booths, gates, police or other emergency personnel, road workers, stopped buses or other vehicles, one-way bridge arbitration, ferry entrances, etc. Thus, the waiting sensor may be utilized to identify potential obstacles and implement one or more controls (e.g., slowing, stopping, etc.) that may not be possible relying solely on the obstacle sensor.

[0064] The map sensor may include mechanisms by which it discerns behavior and, in some examples, determines specific examples of what conventions apply in a particular area. For example, the map sensor may determine from data representing previous driving or travel that: there are no U-turns at a certain intersection between certain hours; electronic signs indicating lane directionality change depending on the time of day; two traffic lights in close proximity (e.g., barely offset from each other) are associated with different roads; in Rhode Island, the first car waiting to turn left at a traffic light violates the law by turning in front of oncoming traffic when the light turns green; and / or other information. The map sensor may notify the vehicle 600 of static or stationary infrastructure objects and obstacles. The map sensor may also generate information for a wait sensor and / or a path sensor, for example, to determine which light at an intersection must be green for the vehicle 600 to take a particular path.

[0065] In some examples, information from the map sensor may be sent, transmitted, and / or provided to a server (e.g., Figure 6D The map manager of the server 678 may be a map sensor and / or a location manager of the vehicle 600, and information from the server may be sent, transmitted, and / or provided to the map sensor and / or location manager of the vehicle 600. The map manager may include a cloud map application that is located remotely from the vehicle 600 and is accessible by the vehicle 600 via one or more networks. For example, the map sensor and / or location manager of the vehicle 600 may communicate with one or more other components or features of the map manager and / or server to inform the map sensor and / or location manager of past and current driving or travel of the vehicle 600 and past and current driving or travel of other vehicles. The map manager may provide a map output (e.g., map data) that can be localized by the location manager based on the specific location of the vehicle 600, and the localized map output may be used by the world model manager to generate and / or update the world model.

[0066] The planning component may include a route planner, a lane planner, a behavior planner, and a behavior selector, among other components, features, and / or functionality. The route planner may use information from the map sensor, the map manager, and / or the positioning manager, as well as other information, to generate a planned path, which may include GNSS waypoints (e.g., GPS waypoints), 3D world coordinates (e.g., Cartesian coordinates, polar coordinates, etc.) indicating coordinates relative to an origin on the vehicle 600, and the like. Waypoints may represent a specific distance from the vehicle 600 into the future, such as a number of city blocks, kilometers, feet, inches, miles, etc., which may be used as a target for the lane planner.

[0067] The lane planner can use a lane map (e.g., a lane map from a path sensor), object poses within the lane map (e.g., according to a localization manager), and / or a target point and direction at a certain distance in the future from a route planner as input. The target point and direction can be mapped to the best matching drivable point and direction in the lane map (e.g., based on GNSS and / or compass direction). A graph search algorithm can then be performed on the lane map from the current edge in the lane map to find the shortest path to the target point.

[0068] The behavior planner can determine the feasibility of basic behaviors of the vehicle 600, such as staying in the lane or changing lanes to the left or right, so that the feasible behaviors can match the most desired behavior output from the lane planner. For example, if the desired behavior is determined to be unsafe and / or unavailable, a default behavior can be selected instead (for example, when the desired behavior or lane change is unsafe, the default behavior may be staying in the lane).

[0069] One or more control components can follow a trajectory or path (lateral and longitudinal) that has been received from the behavior selector of one or more planning components as closely as possible and within the capabilities of the vehicle 600. The control components can use tight feedback to handle unmodeled non-planned events or behaviors and / or anything that causes a difference from the ideal (e.g., unexpected delays). In some examples, one or more control components can use a forward prediction model that takes the control as an input variable and generates a prediction that can be compared to the desired state (e.g., compared to the desired lateral and longitudinal path requested by the one or more planning components). Controls that minimize the difference can be determined.

[0070] One or more obstacle avoidance components can help the autonomous vehicle 600 avoid collisions with objects (e.g., moving and stationary objects). The one or more obstacle avoidance components may include a computational mechanism for obstacle avoidance at a "primitive level" and may act as a "survival brain" or "reptilian brain" for the vehicle 600. In some examples, the one or more obstacle avoidance components may be used independently of the components, features, and / or functions of the vehicle 600 that are required to obey traffic laws and drive courteously. In such an example, the one or more obstacle avoidance components may ignore traffic laws, road rules, and courteous driving norms in order to ensure that no collision occurs between the vehicle 600 and any object. Thus, the obstacle avoidance layer may be a separate layer from the road rules layer, and the obstacle avoidance layer may ensure that the vehicle 600 performs safe actions only from the perspective of obstacle avoidance. On the other hand, the road rules layer may ensure that the vehicle obeys traffic laws and conventions and adheres to legal and conventional rights of way (as described herein).

[0071] In some examples, the drivable path and / or target detection can be used by one or more obstacle avoidance components to determine the control or action to take. For example, the drivable path can provide one or more obstacle avoidance components with an indication of where the vehicle 600 can maneuver without striking any objects, structures, etc., or at least where no static structures exist.

[0072] In a non-limiting embodiment, one or more obstacle avoidance components may be implemented as separate, discrete features of the vehicle 600. For example, one or more obstacle avoidance components may operate separately from (e.g., in parallel with, before, and / or after) the planning layer, the control layer, the actuation layer, and / or other layers of the driving stack 128.

[0073] Thus, the vehicle 600 can use this information (e.g., such as edges or tracks of a path) to navigate, plan, or otherwise perform one or more maneuvers (e.g., lane keeping, lane changing, path planning, merging, splitting, etc.) within the environment.

[0074] Now refer to Figure 5 , each block of the method 500 described herein comprises a computing process that can be performed using any combination of hardware, firmware, and / or software. For example, the various functions may be implemented by a processor executing instructions stored in a memory. The method 500 may also be embodied as computer-usable instructions stored on a computer storage medium. The method 500 may be provided by a standalone application, a service, or a hosted service (standalone or in combination with other hosted services), or a plug-in to another product, to name a few. Furthermore, by way of example, with respect to Figure 1A The method 500 is described with reference to the process 100 of FIG. However, the method 500 may additionally or alternatively be performed by any system or within any process, or any combination of systems and processes, including but not limited to those described herein.

[0075] Figure 5 is a flow chart illustrating a method 500 for computing an agent trajectory using a DNN, according to some embodiments of the present disclosure. At block B 502 , the method 500 includes applying first data representing map information and second data representing a previous location of the agent to a neural network. For example, the map information 112 , one or more past locations 110 , wait conditions 114 , and / or other inputs 108 may be applied to the DNN 116 .

[0076] At block B504, method 500 includes computing, using a neural network, third data representing a first confidence map corresponding to a first future time and second data representing a second confidence map corresponding to a second future time subsequent to the first future time. For example, DNN 116 may compute confidence fields 120 corresponding to the first future time slice and the second future time slice subsequent to the first future time slice.

[0077] At block B506, method 500 includes computing, using the neural network, fourth data representing a vector field corresponding to a second future time. For example, DNN 116 may compute vector field 122 corresponding to the second future time slice.

[0078] At block B508, method 500 includes determining one or more second points on the second confidence map that indicate a second future location of the actor. For example, post-processor 124 may perform clustering, weighted averaging, and / or another technique to determine points indicating the presence of the actor from confidence field 120 corresponding to the second future time slice.

[0079] At block B510 , method 500 includes determining one or more vectors from the vector field corresponding to the one or more second points. For example, vectors from vector field 122 corresponding to points from confidence field 120 may be determined.

[0080] At block B512, method 500 includes determining one or more first points on a first confidence map based at least in part on the one or more vectors. For example, vectors may be utilized to determine potential locations or points on confidence field 120 corresponding to a first future time slice indicating the presence of an actor.

[0081] At block B514, method 500 includes determining a first future location of the agent prior to the second future location based at least in part on the one or more first points. For example, the first future location of the agent may be determined using a location or point from confidence field 120 corresponding to the first future time slice.

[0082] At block B516, method 500 includes generating a predicted future trajectory for the agent using the first future location and the second future location. For example, trajectory generator 126 may generate a trajectory using the predicted locations at the first future time slice and the second future time slice. This trajectory may then be used by driving stack 128 to perform one or more operations, such as, but not limited to, those described herein.

[0083] Training DNN for future trajectory prediction

[0084] See also Figures 1A-1BTo train the DNN 116, a training engine 130 may be employed. The training engine may rely on ground truth data and one or more loss functions to update the weights and parameters of the DNN 116. To determine the ground truth data, training data may first be collected, and ground truth data corresponding thereto may be generated. In some embodiments, event weighting and / or data mining may be performed to collect and / or generate the most effective training data for training the DNN 116. For example, any number of records (which may be large and collected over a long period of driving) may be used to generate a training dataset for training the DNN 116. Typically, the dataset may contain straight, constant velocity segments that, if used directly, may bias the DNN 116 toward predicted motions that do not have higher fidelity than using kinematics alone to make such predictions. Thus, segments of the records used for training data that do not represent straight, constant velocity may be identified as more interesting or nontrivial. In such an example, non-trivial or more interesting training data (e.g., lane changes, curvy roads, overtaking, aggressive maneuvers, etc.) can be weighted more heavily within the loss function than more trivial or less interesting (e.g., straight lines, constant speed, etc.). A triviality or relevance / importance factor can be associated with each training data instance, and this factor can be used to determine the weighting of the loss function relative to the corresponding training data instance. In some embodiments, active learning can be used to enable the DNN 116 to improve its predictive capabilities in non-trivial and more interesting, relevant, and / or important scenarios using one or more labels.

[0085] In some embodiments, the marker can be determined using statistical methods. However, in other embodiments, heuristic algorithms, machine learning and / or other techniques can be implemented to determine the marker. In the case of using statistical methods, various measurements can be calculated to determine the triviality factor of the instance of the training data. For example, but not limited to, the standard deviation of speed, the ratio of the standard deviation in the longitudinal (Y) and lateral (X) directions and / or the change in the standard deviation of speed can represent a marker that performs well in identifying non-trivial training data instances. Larger standard deviations tend to appear in the X and Y directions at intersections that indicate non-trivial movement of traffic. A larger ratio of the standard deviation in the lateral to longitudinal directions tends to indicate overtaking (cut-in) and other urban driving scenarios. In addition, the change in standard deviation during the training data set (e.g., 3-6 seconds) can generally indicate congested traffic scenarios.

[0086] Furthermore, in some embodiments, the output of the automatic label generation pipeline can be utilized to detect the occurrence of rare or non-trivial events, such as a vehicle passing in front of the ego vehicle 600. The automatic label generation pipeline can include generating training data using one or more components of the driving stack 128 in one or more data collection vehicles as the vehicle traverses different environments. For example, the sensors of the data collection vehicle can be calibrated (e.g., by running a self-calibration) to ensure accurate translation between each sensor tracking the motion of the ego vehicle and other agents in the environment. The calibration values ​​can be used by the perception stack (e.g., components of the driving stack 128) to automatically generate labels for training data for the DNN 116. Thus, as the data tracks the vehicle's movement through the environment, the collected data can indicate the agent's position over time. For example, at time T1, an agent can be detected, and to train the DNN 116 to predict the agent's future position, the agent's motion can be tracked after time T1 until some later time T2. Thus, the agent's position from time T1 to time T2 can be used as ground truth data for training the DNN 116 to predict the agent's future position. In such an example, where training data is applied to DNN 116 during training to include the positions of the actors over a period of time up to time T1, the predictions of DNN 116 can be compared with the ground truth data automatically generated by the pipeline using one or more loss functions. This process can be repeated for any number of iterations on any number of training data instances until the accuracy of DNN 116 converges to an acceptable level of accuracy. While this automatic label generation pipeline can be computationally intensive, the burden can be reduced because many processes can be run in parallel using one or more GPUs.

[0087] As described above, the output of the pipeline can be used to detect rare occurrences. For example, the output from the perception stack used by the data collection vehicle and / or the data collected from the data collection vehicle can represent a world model (e.g., the state of static and dynamic actors or elements in the environment). Using heuristic algorithms, statistical models and / or DNNs, these rare events can be mined to build a dataset. For example, a dataset corresponding to a specific maneuver can be established. As an example, with respect to overtaking, heuristic algorithms can be used to define rules such as (1) if the identity (ID) of the current vehicle (CVIP) in the path changes; and (2) if the distance to the new CVIP is less than 80 meters, mark the instance as a potential overtaking. This process can then be repeated across any number of training data instances to identify rare or non-trivial events. Once rare or non-trivial events are identified, they can be sent to a tagger for verification (e.g., to verify whether this is an overtaking). When the DNN 116 is trained to recognize such events, these heuristic algorithms can be replaced or augmented with the DNN 116 to reduce false positive and false negative rates.

[0088] Once the training data and ground truth are generated, collected, and / or received, the DNN 116 can be trained using a loss function. In some instances, a single loss function can be used, while in other examples, more than one loss function can be used. Where more than one loss function is used, a first loss function can be used to train the DNN 116 to more accurately predict the confidence field 120, a second loss function can be used to train the DNN 116 to more accurately predict the vector field 122, and / or a third loss function can be used to determine an overall loss from the first and second loss functions—e.g., using weighting. In such an example, the loss function for the confidence field 120 can include a binary cross entropy loss function and can be defined according to the following equation (5):

[0089] H(p,y)=-∑ i y i log(p i ) (5)

[0090] The loss function of the vector field 122 may include an L1 or L2 norm loss function and may be defined according to the following equation (2):

[0091]

[0092] The total loss function can be calculated as the sum of the first loss (Equation (5)) and the second loss (Equation (6)), weighted according to the following equation (7):

[0093] L=αH(p, y)+βR(v, t) (7)

[0094] where α and β are scalar loss weights and can be chosen empirically.

[0095] Example autonomous vehicle

[0096] Figure 6A6 is an illustration of an example autonomous vehicle 600 according to some embodiments of the present disclosure. Autonomous vehicle 600 (alternatively referred to herein as "vehicle 600") may include, but is not limited to, a passenger vehicle, such as a car, a truck, a bus, a first responder vehicle, a shuttle, an electric or electric bicycle, a motorcycle, a fire truck, a police vehicle, an ambulance, a boat, a construction vehicle, an underwater vehicle, a drone, and / or another type of vehicle (e.g., unmanned and / or accommodating one or more passengers). Autonomous vehicles are generally described according to the levels of automation defined by the National Highway Traffic Safety Administration (NHTSA), a division of the U.S. Department of Transportation, and the Society of Automotive Engineers (SAE), "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles" (Standard No. J3016-201806, issued on June 15, 2018, Standard No. J3016-201609, issued on September 30, 2016, and previous and future versions of the same). The vehicle 600 may be capable of implementing functionality consistent with one or more of autonomous driving levels 3 to 5. For example, depending on the embodiment, the vehicle 600 may be capable of conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5).

[0097] Vehicle 600 may include components such as a chassis, a body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other vehicle components. Vehicle 600 may include a propulsion system 650, such as an internal combustion engine, a hybrid power plant, an all-electric engine, and / or another type of propulsion system. Propulsion system 650 may be connected to a drivetrain of vehicle 600, which may include a transmission, to achieve propulsion of vehicle 600. Propulsion system 650 may be controlled in response to receiving a signal from throttle / accelerator 652.

[0098] A steering system 654, which may include a steering wheel, may be used to steer vehicle 600 (e.g., along a desired path or route) when propulsion system 650 is operating (e.g., when the vehicle is in motion). Steering system 654 may receive signals from steering actuator 656. For fully automated (Level 5) functionality, a steering wheel may be optional.

[0099] Brake sensor system 646 may be used to operate vehicle brakes in response to receiving signals from brake actuator 648 and / or brake sensors.

[0100] May include one or more system on chip (SoC) 604 ( Figure 6C) and / or one or more GPUs can provide signals (e.g., representing commands) to one or more components and / or systems of the vehicle 600. For example, the one or more controllers can send signals to operate the vehicle brakes via one or more brake actuators 648, to operate the steering system 654 via one or more steering actuators 656, and to operate the propulsion system 650 via one or more throttles / accelerators 652. The one or more controllers 636 can include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operational commands (e.g., signals representing commands) to enable autonomous driving and / or assist a human driver in driving the vehicle 600. The one or more controllers 636 can include a first controller 636 for autonomous driving functions, a second controller 636 for functional safety functions, a third controller 636 for artificial intelligence functions (e.g., computer vision), a fourth controller 636 for infotainment functions, a fifth controller 636 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 636 may handle two or more of the above functions, two or more controllers 636 may handle a single function, and / or any combination thereof.

[0101] The one or more controllers 636 may provide signals for controlling one or more components and / or systems of the vehicle 600 in response to sensor data (e.g., sensor inputs) received from one or more sensors. The sensor data may be received from, for example and without limitation, a global navigation satellite system sensor 658 (e.g., a global positioning system sensor), a RADAR sensor 660, an ultrasonic sensor 662, a LIDAR sensor 664, an inertial measurement unit (IMU) sensor 666 (e.g., an accelerometer, a gyroscope, a magnetic compass, a magnetometer, etc.), a microphone 696, a stereo camera 668, a wide-angle camera 670 (e.g., a fisheye camera), an infrared camera 672, a surround camera 674 (e.g., a 360-degree camera), a long-range and / or mid-range camera 698, a speed sensor 644 (e.g., for measuring the velocity of the vehicle 600), a vibration sensor 642, a steering sensor 640, a brake sensor (e.g., as part of a brake sensor system 646), and / or other sensor types.

[0102] One or more of the controllers 636 may receive input (e.g., represented by input data) from the instrument cluster 632 of the vehicle 600 and provide output (e.g., represented by output data, display data, etc.) via a human machine interface (HMI) display 634, an audible annunciator, a speaker, and / or via other components of the vehicle 600. These outputs may include information such as vehicle speed, velocity, time, map data (e.g., Figure 6CThe HMI display 634 may display information such as the HD map 622 of the vehicle 600, location data (e.g., the location of the vehicle 600 on the map), direction, the location of other vehicles (e.g., an occupancy grid), information about objects and object states as sensed by the controller 636, etc. For example, the HMI display 634 may display information about the presence of one or more objects (e.g., street signs, warning signs, traffic light changes, etc.) and / or information about driving maneuvers that the vehicle has made, is making, or will make (e.g., changing lanes now, leaving 34B in two miles, etc.).

[0103] The vehicle 600 also includes a network interface 624 that can communicate over one or more networks using one or more wireless antennas 626 and / or a modem. For example, the network interface 624 can be capable of communicating over LTE, WCDMA, UMTS, GSM, CDMA2000, etc. The one or more wireless antennas 626 can also enable communication between objects in the environment (e.g., vehicles, mobile devices, etc.) using one or more local area networks such as Bluetooth, Bluetooth LE, Z-wave, ZigBee, etc. and / or one or more low power wide area networks (LPWANs) such as LoRaWAN, SigFox, etc.

[0104] Figure 6B For use according to some embodiments of the present disclosure Figure 6A An example of camera positions and fields of view for an example autonomous vehicle 600 is shown. The cameras and respective fields of view are an example embodiment and are not intended to be limiting. For example, additional and / or alternative cameras may be included and / or the cameras may be located at different locations on the vehicle 600.

[0105] The camera type used for the camera may include, but is not limited to, a digital camera that may be suitable for use with components and / or systems of the vehicle 600. The camera may operate at Automotive Safety Integrity Level (ASIL) B and / or at another ASIL. The camera type may have any image capture rate, such as 60 frames per second (fps), 620fps, 240fps, and the like, depending on the embodiment. The camera may be capable of using a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include a red-white-white-white (RCCC) color filter array, a red-white-white-blue (RCCB) color filter array, a red-blue-green-white (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, a clear pixel camera such as a camera with an RCCC, RCCB, and / or RBGC color filter array may be used in an effort to improve light sensitivity.

[0106] In some examples, one or more of the cameras can be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function monocular camera can be installed to provide functions including lane departure warning, traffic sign assistance, and intelligent headlight control. One or more of the cameras (e.g., all of the cameras) can simultaneously record and provide image data (e.g., video).

[0107] One or more of the cameras can be mounted in a mounting assembly, such as a custom-designed (3-D printed) assembly, to cut off stray light and reflections from within the car (e.g., reflections from the dashboard reflected in the windshield mirror) that could interfere with the camera's ability to capture image data. With respect to the wing mirror mounting assembly, the wing mirror assembly can be custom 3-D printed so that the camera mounting plate matches the shape of the wing mirror. In some examples, one or more cameras can be integrated into the wing mirror. For side-view cameras, one or more cameras can also be integrated into the four pillars at each corner of the cab.

[0108] A camera (e.g., a front-facing camera) having a field of view that includes a portion of the environment in front of the vehicle 600 can be used for surround vision to help identify the forward path and obstacles, as well as assist in providing information critical to generating an occupancy grid and / or determining a preferred vehicle path with the help of one or more controllers 636 and / or control SoCs. The front-facing camera can be used to perform many of the same ADAS functions as LIDAR, including emergency braking, pedestrian detection, and collision avoidance. The front-facing camera can also be used for ADAS functions and systems, including lane departure warning ("LDW"), autonomous cruise control ("ACC"), and / or other functions such as traffic sign recognition.

[0109] A variety of cameras can be used in the front-facing configuration, including, for example, a monocular camera platform including a CMOS (complementary metal oxide semiconductor) color imager. Another example could be a wide-angle camera 670, which can be used to sense objects entering the field of view from the periphery (e.g., pedestrians, intersection traffic, or bicycles). Although Figure 6B The figure shows only one wide-angle camera, but there can be any number of wide-angle cameras 670 on the vehicle 600. In addition, long-range cameras 698 (e.g., a long-view stereo camera pair) can be used for depth-based object detection, especially for objects for which neural networks have not yet been trained. Long-range cameras 698 can also be used for object detection and classification and basic object tracking.

[0110] One or more stereo cameras 668 may also be included in the front configuration. The stereo camera 668 may include an integrated control unit including an expandable processing unit that may provide a multi-core microprocessor and programmable logic (FPGA) with an integrated CAN or Ethernet interface on a single chip. Such a unit may be used to generate a 3-D map of the vehicle environment, including distance estimates for all points in the image. An alternative stereo camera 668 may include a compact stereo vision sensor that may include two camera lenses (one on the left and one on the right) and an image processing chip that may measure the distance from the vehicle to the target object and use the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 668 may be used in addition to or alternatively to those described herein.

[0111] Cameras with a field of view that includes portions of the environment to the sides of the vehicle 600 (e.g., side-view cameras) can be used for surround viewing, providing information used to create and update occupancy grids and generate side impact collision warnings. For example, surround cameras 674 (e.g., Figure 6B Four surround cameras 674 (shown in FIG) can be placed on the vehicle 600. The surround cameras 674 can include a wide-angle camera 670, a fisheye camera, a 360-degree camera, and / or the like. For example, the four fisheye cameras can be placed on the front, rear, and sides of the vehicle. In an alternative arrangement, the vehicle can use three surround cameras 674 (e.g., left, right, and rear), and can utilize one or more other cameras (e.g., a forward-facing camera) as a fourth surround-view camera.

[0112] A camera having a field of view that includes a portion of the environment behind the vehicle 600 (e.g., a rearview camera) can be used to assist with parking, surround view, rear collision warning, and creating and updating occupancy grids. A variety of cameras can be used, including but not limited to cameras that are also suitable as front-facing cameras as described herein (e.g., long-range and / or mid-range cameras 698, stereo cameras 668, infrared cameras 672, etc.).

[0113] Figure 6C For use according to some embodiments of the present disclosure Figure 6A6. Block diagram of an example system architecture for an example autonomous vehicle 600. It will be understood that this arrangement and other arrangements described herein are set forth merely as examples. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely. Further, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in conjunction with other components, and in any appropriate combination and location. The various functions described herein as being performed by entities may be implemented by hardware, firmware, and / or software. For example, the various functions may be implemented by a processor executing instructions stored in memory.

[0114] Figure 6C Each of the components, features, and systems of vehicle 600 is illustrated as being connected via bus 602. Bus 602 may include a controller area network (CAN) data interface (alternatively, referred to herein as a "CAN bus"). CAN may be a network internal to vehicle 600 that assists in controlling various features and functions of vehicle 600, such as actuation of brakes, acceleration, braking, steering, windshield wipers, and the like. The CAN bus may be configured to have tens or even hundreds of nodes, each with its own unique identifier (e.g., a CAN ID). The CAN bus may be read to find steering wheel angle, ground speed, engine revolutions per minute (RPM), button positions, and / or other vehicle status indicators. The CAN bus may be ASIL B compliant.

[0115] Although bus 602 is described here as a CAN bus, this is not intended to be limiting. For example, FlexRay and / or Ethernet may be used in addition to or in lieu of a CAN bus. Furthermore, although bus 602 is represented by a single line, this is not intended to be limiting. For example, there may be any number of buses 602, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 602 may be used to perform different functions and / or may be used for redundancy. For example, a first bus 602 may be used for collision avoidance functionality, and a second bus 602 may be used for drive control. In any example, each bus 602 may communicate with any component of vehicle 600, and two or more buses 602 may communicate with the same component. In some examples, each SoC 604, each controller 636, and / or each computer within the vehicle may have access to the same input data (e.g., input from sensors on vehicle 600) and may be connected to a common bus such as a CAN bus.

[0116] The vehicle 600 may include one or more controllers 636, such as those described herein. Figure 6A Controller 636 may be used for a variety of functions. Controller 636 may be coupled to any of the various components and systems of vehicle 600 and may be used for control of vehicle 600, artificial intelligence of vehicle 600, infotainment for vehicle 600, and / or the like.

[0117] The vehicle 600 may include one or more system on a chip (SoC) 604. The SoC 604 may include a CPU 606, a GPU 608, a processor 610, a cache 612, an accelerator 614, a data store 616, and / or other components and features not shown. The SoC 604 may be used to control the vehicle 600 in a variety of platforms and systems. For example, the one or more SoCs 604 may be combined with an HD map 622 in a system (e.g., a system of the vehicle 600), which may be downloaded from one or more servers (e.g., a server) via a network interface 624. Figure 6D one or more servers 678) to obtain map refreshes and / or updates.

[0118] The CPU 606 may include a CPU cluster or CPU complex (alternatively, referred to herein as a "CCPLEX"). The CPU 606 may include multiple cores and / or L2 caches. For example, in some embodiments, the CPU 606 may include eight cores in a coherent multiprocessor configuration. In some embodiments, the CPU 606 may include four dual-core clusters, each with a dedicated L2 cache (e.g., a 2MB L2 cache). The CPU 606 (e.g., CCPLEX) may be configured to support simultaneous cluster operations such that any combination of CPU 606 clusters can be active at any given time.

[0119] The CPU 606 may implement power management capabilities including one or more of the following features: each hardware block may be automatically clock gated when idle to conserve dynamic power; each core clock may be gated when the core is not actively executing instructions due to the execution of WFI / WFE instructions; each core may be independently power gated; each core cluster may be independently clock gated when all cores are clock gated or power gated; and / or each core cluster may be independently power gated when all cores are power gated. The CPU 606 may further implement an enhanced algorithm for managing power states, in which allowed power states and expected wakeup times are specified, and hardware / microcode determines the optimal power state to enter for the core, cluster, and CCPLEX. The processing core may support a simplified power state entry sequence in software, with this work being offloaded to the microcode.

[0120] The GPU 608 may include an integrated GPU (alternatively referred to herein as an "iGPU"). The GPU 608 may be programmable and efficient for parallel workloads. In some examples, the GPU 608 may use an enhanced tensor instruction set. The GPU 608 may include one or more streaming microprocessors, each of which may include an L1 cache (e.g., an L1 cache with at least 96KB of storage capacity), and two or more of these streaming microprocessors may share an L2 cache (e.g., an L2 cache with 512KB of storage capacity). In some embodiments, the GPU 608 may include at least eight streaming microprocessors. The GPU 608 may use a computing application programming interface (API). In addition, the GPU 608 may use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0121] In the case of automotive and embedded use, GPU 608 can be power optimized to achieve optimal performance. For example, GPU 608 can be manufactured on fin field effect transistors (FinFETs). However, this is not intended to be limiting, and GPU 608 can be manufactured using other semiconductor manufacturing processes. Each streaming microprocessor can merge several mixed precision processing cores divided into multiple blocks. For example and without limitation, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed precision NVIDIA tensor cores for deep learning matrix arithmetic, L0 instruction cache, warp scheduler, dispatch unit and / or 64KB register file. In addition, the streaming microprocessor may include independent parallel integer and floating point data paths to provide efficient execution of workloads using a mix of computation and addressing calculations. The streaming microprocessor may include independent thread scheduling capabilities to allow for finer-grained synchronization and collaboration between parallel threads. Streaming microprocessors may include a combined L1 data cache and shared memory unit to increase performance while simplifying programming.

[0122] The GPU 608 can include high bandwidth memory (HBM) and / or a 16GB HBM2 memory subsystem that provides a peak memory bandwidth of approximately 900 GB / s in some examples. In some examples, synchronous graphics random access memory (SGRAM), such as fifth generation graphics double data rate synchronous random access memory (GDDR5), can be used in addition to or in lieu of HBM memory.

[0123] The GPU 608 may include unified memory technology that includes access counters to allow memory pages to be more accurately migrated to the processor that accesses them most frequently, thereby improving the efficiency of memory ranges shared between processors. In some examples, address translation services (ATS) support may be used to allow the GPU 608 to directly access the CPU 606 page tables. In such an example, when the GPU 608 memory management unit (MMU) experiences a miss, an address translation request may be transmitted to the CPU 606. In response, the CPU 606 may look up the virtual-to-physical mapping for the address in its page table and transmit the translation back to the GPU 608. In this way, unified memory technology may allow a single unified virtual address space to be used for memory of both the CPU 606 and the GPU 608, thereby simplifying GPU 608 programming and porting applications to the GPU 608.

[0124] In addition, GPU 608 can include access counters that can track how often GPU 608 accesses the memory of other processors. Access counters can help ensure that memory pages are moved to the physical memory of the processor that accesses them most frequently.

[0125] SoC 604 may include any number of caches 612, including those described herein. For example, cache 612 may include an L3 cache available to both CPU 606 and GPU 608 (e.g., connected to both CPU 606 and GPU 608). Cache 612 may include a write-back cache that can track the state of lines, for example, using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). Depending on the embodiment, the L3 cache may include 4MB or more, although smaller cache sizes may also be used.

[0126] SoC 604 may include one or more accelerators 614 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, SoC 604 may include a hardware acceleration cluster, which may include optimized hardware accelerators and / or large on-chip memory. The large on-chip memory (e.g., 4MB SRAM) may enable the hardware acceleration cluster to accelerate neural networks and other calculations. The hardware acceleration cluster may be used to supplement GPU 608 and offload some tasks of GPU 608 (e.g., freeing up more cycles of GPU 608 for performing other tasks). As an example, accelerator 614 may be used for targeted workloads (e.g., perception, convolutional neural networks (CNNs), etc.) that are sufficiently stable to be easily controlled for acceleration. When used herein, the term "CNN" may include all types of CNNs, including region-based or regional convolutional neural networks (RCNNs) and fast RCNNs (e.g., for object detection).

[0127] The accelerator 614 (e.g., a hardware acceleration cluster) may include a deep learning accelerator (DLA). The DLA may include one or more tensor processing units (TPUs) that can be configured to provide an additional 10 trillion operations per second for deep learning applications and reasoning. The TPU may be an accelerator configured to perform image processing functions (e.g., for CNN, RCNN, etc.) and optimized for performing image processing functions. The DLA may be further optimized for a specific set of neural network types and floating-point operations and reasoning. The design of the DLA may provide higher performance per millimeter than a general-purpose GPU and far exceed the performance of the CPU. The TPU may perform several functions, including a single-instance convolution function, support for INT8, INT16, and FP16 data types for both features and weights, and post-processor functions.

[0128] DLA can quickly and efficiently execute neural networks, particularly CNNs, on processed or unprocessed data for any of a wide variety of functions, such as, but not limited to: CNNs for object recognition and detection using data from camera sensors; CNNs for distance estimation using data from camera sensors; CNNs for emergency vehicle detection and recognition and detection using data from microphones; CNNs for facial recognition and vehicle owner identification using data from camera sensors; and / or CNNs for safety and / or security-related events.

[0129] The DLA can perform any function of the GPU 608, and by using an inference accelerator, for example, the designer can target any function to either the DLA or the GPU 608. For example, the designer can focus the processing of CNNs and floating-point operations on the DLA and leave other functions to the GPU 608 and / or other accelerators 614.

[0130] The accelerator 614 (e.g., a hardware acceleration cluster) may include a programmable vision accelerator (PVA), which may be alternatively referred to herein as a computer vision accelerator. The PVA may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. The PVA may provide a balance between performance and flexibility. For example, each PVA may include, for example and without limitation, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors.

[0131] The RISC core can interact with an image sensor (e.g., an image sensor of any camera described herein), an image signal processor, and / or the like. Each of these RISC cores can include any amount of memory. Depending on the embodiment, the RISC core can use any of a number of protocols. In some examples, the RISC core can execute a real-time operating system (RTOS). The RISC core can be implemented using one or more integrated circuit devices, application specific integrated circuits (ASICs), and / or memory devices. For example, the RISC core can include an instruction cache and / or tightly coupled RAM.

[0132] The DMA can enable components of the PVA to access system memory independently of the CPU 606. The DMA can support any number of features used to provide optimizations for the PVA, including but not limited to support for multi-dimensional addressing and / or circular addressing. In some examples, the DMA can support addressing in up to six or more dimensions, which can include block width, block height, block depth, horizontal block stepping, vertical block stepping, and / or depth stepping.

[0133] A vector processor can be a programmable processor that can be designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing capabilities. In some examples, the PVA can include a PVA core and two vector processing subsystem partitions. The PVA core can include a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem can operate as the main processing engine of the PVA and can include a vector processing unit (VPU), an instruction cache, and / or a vector memory (e.g., VMEM). The VPU core can include a digital signal processor, such as, for example, a single instruction multiple data (SIMD), a very long instruction word (VLIW) digital signal processor. The combination of SIMD and VLIW can enhance throughput and speed.

[0134] Each of the vector processors can include an instruction cache and can be coupled to dedicated memory. As a result, in some examples, each of the vector processors can be configured to execute independently of the other vector processors. In other examples, the vector processors included in a particular PVA can be configured to employ data parallelism. For example, in some embodiments, multiple vector processors included in a single PVA can execute the same computer vision algorithm, but on different regions of an image. In other examples, the vector processors included in a particular PVA can execute different computer vision algorithms simultaneously on the same image, or even execute different algorithms on sequential images or portions of images. Among other things, any number of PVAs can be included in a hardware acceleration cluster, and any number of vector processors can be included in each of these PVAs. In addition, the PVAs can include additional error correction code (ECC) memory to enhance overall system security.

[0135] The accelerator 614 (e.g., a hardware acceleration cluster) may include an on-chip computer vision network and SRAM to provide high bandwidth, low latency SRAM for the accelerator 614. In some examples, the on-chip memory may include at least 4MB of SRAM consisting of, for example and without limitation, eight field-configurable memory blocks that can be accessed by both the PVA and the DLA. Each pair of memory blocks may include an advanced peripheral bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory may be used. The PVA and DLA may access the memory via a backbone that provides high-speed memory access to the PVA and DLA. The backbone may include an on-chip computer vision network that interconnects the PVA and DLA to the memory (e.g., using APB).

[0136] The on-chip computer vision network can include an interface that ensures that both the PVA and DLA provide ready and valid signals before transmitting any control signals / addresses / data. Such an interface can provide separate phases and separate channels for transmitting control signals / addresses / data, as well as burst-based communication for continuous data transmission. This type of interface can comply with ISO 26262 or IEC 61508 standards, but other standards and protocols can also be used.

[0137] In some examples, SoC 604 may include a real-time ray tracing hardware accelerator, such as that described in U.S. patent application Ser. No. 16 / 101,232, filed on Aug. 10, 2018. The real-time ray tracing hardware accelerator may be used to quickly and efficiently determine the position and extent of objects (e.g., within a world model) to generate real-time visual simulations for RADAR signal interpretation, for sound propagation synthesis and / or analysis, for SONAR system simulation, for general wave propagation simulation, for comparison with LIDAR data for positioning and / or other functions, and / or for other uses.

[0138] The accelerator 614 (e.g., a hardware accelerator cluster) has a wide range of uses in autonomous driving. The PVA can be a programmable vision accelerator that can be used in key processing stages in ADAS and autonomous vehicles. The capabilities of the PVA are a good match for algorithmic domains that require predictable processing, low power, and low latency. In other words, the PVA performs well on semi-intensive or intensive rule computations, and even on small data sets that require predictable runtimes with low latency and low power. Therefore, in the context of a platform for autonomous vehicles, the PVA is designed to run classic computer vision algorithms because they are efficient at object detection and integer math operations.

[0139] For example, according to one embodiment of the technology, PVA is used to perform computer stereo vision. In some examples, a semi-global matching-based algorithm can be used, but this is not intended to be limiting. Many applications for Level 3-5 autonomous driving require on-the-fly motion estimation / stereo matching (e.g., structure from motion, pedestrian recognition, lane detection, etc.). PVA can perform computer stereo vision functions on input from two monocular cameras.

[0140] In some examples, PVA can be used to perform dense optical flow, by processing raw RADAR data (e.g., using a 4D Fast Fourier Transform) to provide processed RADAR. In other examples, PVA is used for time-of-flight depth processing, by processing raw time-of-flight data to provide processed time-of-flight data.

[0141] The DLA can be used to run any type of network to enhance control and driving safety, including, for example, a neural network that outputs a confidence measure for each object detection. Such confidence values ​​can be interpreted as probabilities, or as providing a relative "weight" of each detection compared to other detections. This confidence value enables the system to make further decisions about which detections should be considered true positives versus false positives. For example, the system can set a threshold for confidence and only consider detections that exceed the threshold as true positives. In an automatic emergency braking (AEB) system, a false positive detection could cause the vehicle to automatically apply emergency braking, which is clearly undesirable. Therefore, only the most confident detections should be considered triggers for AEB. The DLA can run a neural network to regress the confidence value. This neural network can take as its input at least some subset of parameters, such as bounding box dimensions, a ground plane estimate obtained (e.g., from another subsystem), inertial measurement unit (IMU) sensor 666 output related to the orientation and distance of the vehicle 600, and 3D position estimates of objects obtained from the neural network and / or other sensors (e.g., LIDAR sensor 664 or RADAR sensor 660).

[0142] SoC 604 may include one or more data stores 616 (e.g., memory). Data stores 616 may be on-chip memory of SoC 604 that may store neural networks to be executed on the GPU and / or DLA. In some examples, for redundancy and safety, data stores 616 may be large enough to store multiple instances of the neural network. Data stores 612 may include L2 or L3 cache 612. References to data stores 616 may include references to memory associated with the PVA, DLA, and / or other accelerators 614 as described herein.

[0143] SoC 604 may include one or more processors 610 (e.g., embedded processors). Processor 610 may include a boot and power management processor, which may be a dedicated processor and subsystem for handling boot power and management functions and related safety implementations. The boot and power management processor may be part of the SoC 604 boot sequence and may provide runtime power management services. The boot power and management processor may provide clock and voltage programming, auxiliary system low power state transitions, SoC 604 thermal and temperature sensor management, and / or SoC 604 power state management. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to temperature, and SoC 604 may use the ring oscillator to detect the temperature of CPU 606, GPU 608, and / or accelerator 614. If it is determined that the temperature exceeds a threshold, the boot and power management processor may enter a temperature fault routine and place SoC 604 in a lower power state and / or place vehicle 600 in a driver safety parking mode (e.g., to safely park vehicle 600).

[0144] The processor 610 may also include a set of embedded processors that can be used as an audio processing engine. The audio processing engine can be an audio subsystem that allows for full hardware support for multi-channel audio through multiple interfaces and a wide range of flexible audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core having a digital signal processor with dedicated RAM.

[0145] The processor 610 may also include an always-on processor engine that can provide the necessary hardware features to support low-power sensor management and wake-up use cases. The always-on processor engine may include a processor core, tightly coupled RAM, supporting peripherals (such as timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0146] Processor 610 may also include a safety cluster engine, which includes a dedicated processor subsystem that handles safety management of automotive applications. The safety cluster engine may include two or more processor cores, tightly coupled RAM, supporting peripherals (such as timers, interrupt controllers, etc.), and / or routing logic. In safety mode, the two or more cores can operate in lockstep mode and act as a single core with comparison logic to detect any differences between their operations.

[0147] Processor 610 may also include a real-time camera engine, which may include a dedicated processor subsystem for handling real-time camera management.

[0148] Processor 610 may also include a high dynamic range signal processor, which may include an image signal processor, which is a hardware engine that is part of the camera processing pipeline.

[0149] The processor 610 may include a video image compositer, which may be a processing block (e.g., implemented on a microprocessor) that implements the video post-processing functions required by the video playback application to produce the final image for the player window. The video image compositer may perform lens distortion correction for the wide-angle camera 670, the surround camera 674, and / or for the in-cab monitoring camera sensor. The in-cab monitoring camera sensor is preferably monitored by a neural network running on another instance of the advanced SoC, configured to recognize in-cab events and respond accordingly. The in-cab system may perform lip reading to activate mobile phone service and place calls, dictate emails, change vehicle destinations, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. Certain functions are only available to the driver when the vehicle is operating in autonomous mode and are disabled in other circumstances.

[0150] The video image compositer can include enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, in the presence of motion in the video, the noise reduction appropriately weights spatial information and downweights information provided by neighboring frames. In the case where an image or portion of an image does not include motion, the temporal noise reduction performed by the video image compositer can use information from previous images to reduce noise in the current image.

[0151] The video image compositor can also be configured to perform stereo rectification on the input stereo footage frames. The video image compositor can further be used for user interface composition when the operating system desktop is in use and the GPU 608 does not need to continuously render new surfaces. Even when the GPU 608 is powered on and active for 3D rendering, the video image compositor can be used to offload the GPU 608 to improve performance and responsiveness.

[0152] The SoC 604 may also include a Mobile Industry Processor Interface (MIPI) camera serial interface, a high-speed interface for receiving video and input from a camera, and / or a video input block that may be used for camera and related pixel input functions. The SoC 604 may also include an input / output controller that may be controlled by software and may be used to receive I / O signals that are not assigned to a specific role.

[0153] The SoC 604 may also include a wide range of peripheral interfaces to enable communication with peripherals, audio codecs, power management, and / or other devices. The SoC 604 may be used to process data from cameras (connected via Gigabit multimedia serial links and Ethernet), sensors (e.g., LIDAR sensor 664, RADAR sensor 660, etc., which may be connected via Ethernet), data from the bus 602 (e.g., vehicle 600 speed, steering wheel position, etc.), and data from the GNSS sensor 658 (connected via Ethernet or a CAN bus). The SoC 604 may also include dedicated high-performance mass storage controllers, which may include their own DMA engines and which may be used to free the CPU 606 from routine data management tasks.

[0154] SoC 604 can be an end-to-end platform with a flexible architecture that spans levels 3-5 of automation, providing a comprehensive functional safety architecture that leverages and efficiently uses computer vision and ADAS technologies for diversity and redundancy, along with deep learning tools to provide a platform for a flexible and reliable driving or machine software stack. SoC 604 can be faster, more reliable, and even more energy- and space-efficient than conventional systems. For example, when combined with CPU 606, GPU 608, and data storage 616, accelerator 614 can provide a fast and efficient platform for level 3-5 autonomous vehicles.

[0155] This technology therefore provides capabilities and functionality that cannot be achieved with conventional systems. For example, computer vision algorithms can be executed on CPUs, which can be configured using high-level programming languages ​​such as the C programming language to execute a wide variety of processing algorithms across a wide variety of visual data. However, CPUs often fail to meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. In particular, many CPUs are unable to execute complex object detection algorithms in real time, a requirement for in-vehicle ADAS applications and practical Level 3-5 autonomous vehicles.

[0156] In contrast to conventional systems, by providing a CPU complex, a GPU complex, and a hardware acceleration cluster, the technology described herein allows multiple neural networks to be executed simultaneously and / or sequentially, and the results to be combined to achieve Level 3-5 autonomous driving capabilities. For example, a CNN executed on a DLA or dGPU (e.g., GPU 620) can include text and word recognition, allowing the supercomputer to read and understand traffic signs, including signs for which a neural network has not been specifically trained. The DLA can also include a neural network that can recognize, interpret, and provide semantic understanding of the signs, and pass that semantic understanding to a path planning module running on the CPU complex.

[0157] As another example, as required for Level 3, 4, or 5 driving, multiple neural networks can be run simultaneously. For example, a warning sign consisting of "Caution: Flashing lights indicate icing conditions" along with a light can be interpreted by several neural networks, either independently or collectively. The sign itself can be identified as a traffic sign by a first neural network deployed (e.g., a trained neural network), and the text "Flashing lights indicate icing conditions" can be interpreted by a second neural network deployed, which informs the vehicle's path planning software (preferably executing on a CPU complex) that icing conditions exist when the flashing lights are detected. The flashing lights can be identified by operating a third neural network deployed over multiple frames, which informs the vehicle's path planning software of the presence (or absence) of the flashing lights. All three neural networks can run simultaneously, for example, within the DLA and / or on GPU 608.

[0158] In some examples, a CNN for facial recognition and owner recognition can use data from a camera sensor to identify the presence of an authorized driver and / or owner of vehicle 600. An always-on sensor processing engine can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and in security mode, disable the vehicle when the owner leaves the vehicle. In this way, SoC 604 provides security against theft and / or carjacking.

[0159] In another example, a CNN for emergency vehicle detection and identification can use data from microphone 696 to detect and identify emergency vehicle sirens. In contrast to conventional systems that use general classifiers to detect sirens and manually extract features, SoC 604 uses CNNs to classify environmental and urban sounds and to classify visual data. In a preferred embodiment, the CNN running on the DLA is trained to identify the relative closing speed of emergency vehicles (for example, by using the Doppler effect). The CNN can also be trained to identify emergency vehicles specific to the local area in which the vehicle is operating, as identified by the GNSS sensor 658. Thus, for example, when operating in Europe, the CNN will seek to detect European sirens, and when in the United States, the CNN will seek to identify only North American sirens. Once an emergency vehicle is detected, with the assistance of the ultrasonic sensor 662, the control program can be used to execute emergency vehicle safety routines, slowing the vehicle, pulling to the side of the road, stopping the vehicle, and / or idling the vehicle until the emergency vehicle passes.

[0160] The vehicle may include a CPU 618 (e.g., a discrete CPU or dCPU) that may be coupled to the SoC 604 via a high-speed interconnect (e.g., PCIe). The CPU 618 may include, for example, an X86 processor. The CPU 618 may be used to perform any of a variety of functions, including, for example, arbitrating potentially inconsistent results between ADAS sensors and the SoC 604, and / or monitoring the status and health of the controller 636 and / or the infotainment SoC 630.

[0161] The vehicle 600 may include a GPU 620 (e.g., a discrete GPU or dGPU) that may be coupled to the SoC 604 via a high-speed interconnect (e.g., NVIDIA's NVLINK). The GPU 620 may provide additional artificial intelligence functionality, for example, by executing redundant and / or different neural networks, and may be used to train and / or update the neural network based at least in part on input from sensors of the vehicle 600 (e.g., sensor data).

[0162] The vehicle 600 may also include a network interface 624, which may include one or more wireless antennas 626 (e.g., one or more wireless antennas for different communication protocols, such as a cellular antenna, a Bluetooth antenna, etc.). The network interface 624 can be used to enable wireless connections to the cloud (e.g., to a server 678 and / or other network devices), to other vehicles, and / or to computing devices (e.g., a passenger's client device) via the Internet. In order to communicate with other vehicles, a direct link can be established between the two vehicles, and / or an indirect link can be established (e.g., across a network and through the Internet). The direct link can be provided using a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link can provide the vehicle 600 with information about vehicles approaching the vehicle 600 (e.g., vehicles in front of, to the side of, and / or behind the vehicle 600). This functionality can be part of the cooperative adaptive cruise control functionality of the vehicle 600.

[0163] The network interface 624 may include a SoC that provides modulation and demodulation functions and enables the controller 636 to communicate over a wireless network. The network interface 624 may include an RF front-end for up-conversion from baseband to radio frequency and down-conversion from radio frequency to baseband. The frequency conversion may be performed by a well-known process and / or may be performed using a super-heterodyne process. In some examples, the RF front-end function may be provided by a separate chip. The network interface may include wireless functions for communicating over LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0164] The vehicle 600 may also include data storage 628, which may include off-chip storage (e.g., outside the SoC 604). The data storage 628 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash memory, a hard disk, and / or other components and / or devices that can store at least one bit of data.

[0165] The vehicle 600 may also include a GNSS sensor 658. The GNSS sensor 658 (e.g., GPS and / or assisted GPS sensor) is used to assist with mapping, perception, occupancy grid generation, and / or path planning functions. Any number of GNSS sensors 658 may be used, including, for example and without limitation, a GPS using a USB connector with an Ethernet to serial (RS-232) bridge.

[0166] The vehicle 600 may also include a RADAR sensor 660. The RADAR sensor 660 may be used by the vehicle 600 for remote vehicle detection even in darkness and / or in adverse weather conditions. The RADAR functional safety level may be ASIL B. The RADAR sensor 660 may use CAN and / or bus 602 (e.g., to transmit data generated by the RADAR sensor 660) for control and access to object tracking data, and in some examples access Ethernet to access raw data. A variety of RADAR sensor types may be used. For example and without limitation, the RADAR sensor 660 may be suitable for front, rear, and side RADAR use. In some examples, a pulsed Doppler RADAR sensor is used.

[0167] The RADAR sensor 660 can include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range side coverage, and so on. In some examples, the long-range RADAR can be used for adaptive cruise control functions. The long-range RADAR system can provide a wide field of view (e.g., within 250m) achieved by two or more independent scans. The RADAR sensor 660 can help distinguish between static objects and moving objects and can be used by the ADAS system for emergency braking assistance and forward collision warnings. The long-range RADAR sensor may include a single-station multimode RADAR with multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In the example with six antennas, the central four antennas can create a focused beam pattern that is designed to record the surroundings of the vehicle 600 at a higher rate with minimal traffic interference from adjacent lanes. The other two antennas can expand the field of view, making it possible to quickly detect vehicles entering or leaving the lane of the vehicle 600.

[0168] As an example, a medium-range RADAR system may include a range of up to 660m (front) or 80m (rear) and a field of view of up to 42 degrees (front) or 650 degrees (rear). A short-range RADAR system may include, but is not limited to, a RADAR sensor designed to be mounted on both ends of the rear bumper. When mounted on both ends of the rear bumper, such a RADAR sensor system can create two beams that continuously monitor the blind spots behind and beside the vehicle.

[0169] Short-range RADAR systems can be used in ADAS systems for blind spot detection and / or lane change assistance.

[0170] Vehicle 600 may also include ultrasonic sensors 662. Ultrasonic sensors 662, which may be located on the front, rear, and / or sides of vehicle 600, may be used for parking assistance and / or for creating and updating an occupancy grid. A variety of ultrasonic sensors 662 may be used, and different ultrasonic sensors 662 may have different detection ranges (e.g., 2.5 m, 4 m). Ultrasonic sensors 662 may operate at functional safety level ASIL B.

[0171] Vehicle 600 may include a LIDAR sensor 664. LIDAR sensor 664 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. LIDAR sensor 664 may be ASIL B functional safety level. In some examples, vehicle 600 may include multiple LIDAR sensors 664 (e.g., two, four, six, etc.) that may use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0172] In some examples, the LIDAR sensor 664 may be capable of providing a list of objects and their distances for a 360-degree field of view. Commercially available LIDAR sensors 664 may have, for example, an advertised range of approximately 600 meters, an accuracy of 2-3 cm, and support for 600 Mbps Ethernet connections. In some examples, one or more non-obtrusive LIDAR sensors 664 may be used. In such examples, the LIDAR sensor 664 may be implemented as a small device that can be embedded in the front, back, sides, and / or corners of the vehicle 600. In such an example, the LIDAR sensor 664 may provide a field of view of up to 620 degrees horizontally and 35 degrees vertically, with a range of 200 meters, even for low-reflectivity objects. The front-mounted LIDAR sensor 664 may be configured for a horizontal field of view between 45 and 135 degrees.

[0173] In some examples, LIDAR technologies such as 3D flash LIDAR may also be used. 3D flash LIDAR uses flashes of laser light as an emission source to illuminate the vehicle's surroundings up to about 200 m. The flash LIDAR unit includes a receiver that records the laser pulse transmission time and reflected light on each pixel, which in turn corresponds to the range from the vehicle to the object. Flash LIDAR can allow a highly accurate and distortion-free image of the surrounding environment to be generated with each laser flash. In some examples, four flash LIDAR sensors may be deployed, one on each side of the vehicle 600. Available 3D flash LIDAR systems include solid-state 3D staring array LIDAR cameras (e.g., non-scanning LIDAR devices) with no moving parts other than a fan. The flash LIDAR device may use 5 nanosecond Class I (eye-safe) laser pulses per frame and may capture the reflected laser light in the form of a 3D range point cloud and co-registered intensity data. By using flash LIDAR, and because flash LIDAR is a solid-state device with no moving parts, the LIDAR sensor 664 may be less susceptible to motion blur, vibration, and / or shock.

[0174] The vehicle may also include an IMU sensor 666. In some examples, the IMU sensor 666 may be located at the center of the rear axle of the vehicle 600. The IMU sensor 666 may include, for example and without limitation, an accelerometer, a magnetometer, a gyroscope, a magnetic compass, and / or other sensor types. In some examples, such as in a six-axis application, the IMU sensor 666 may include an accelerometer and a gyroscope, while in a nine-axis application, the IMU sensor 666 may include an accelerometer, a gyroscope, and a magnetometer.

[0175] In some embodiments, the IMU sensor 666 can be implemented as a miniature, high-performance GPS-assisted inertial navigation system (GPS / INS) that combines micro-electromechanical systems (MEMS) inertial sensors, a high-sensitivity GPS receiver, and advanced Kalman filtering algorithms to provide estimates of position, velocity, and attitude. Thus, in some examples, the IMU sensor 666 can enable the vehicle 600 to estimate heading without input from a magnetic sensor by directly observing and correlating velocity changes from the GPS to the IMU sensor 666. In some examples, the IMU sensor 666 and the GNSS sensor 658 can be combined into a single integrated unit.

[0176] The vehicle may include microphones 696 positioned in and / or around the vehicle 600. The microphones 696 may be used for, among other things, emergency vehicle detection and identification.

[0177] The vehicle may also include any number of camera types, including stereo cameras 668, wide angle cameras 670, infrared cameras 672, surround cameras 674, long and / or medium range cameras 698, and / or other camera types. These cameras may be used to capture image data around the entire periphery of the vehicle 600. The type of camera used depends on the embodiment and the requirements of the vehicle 600, and any combination of camera types may be used to provide the necessary coverage around the vehicle 600. Additionally, the number of cameras may vary depending on the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. As an example and not limitation, the cameras may support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet. Each of the cameras described herein may include a 10GbE GPIO (100GbE) ... Figure 6A and Figure 6B Described in more detail.

[0178] Vehicle 600 may also include a vibration sensor 642. Vibration sensor 642 can measure the vibration of a component of the vehicle, such as an axle. For example, a change in vibration can indicate a change in the road surface. In another example, when two or more vibration sensors 642 are used, the difference between the vibrations can be used to determine friction or slip on the road surface (e.g., when there is a vibration difference between a powered drive shaft and a freely rotating shaft).

[0179] The vehicle 600 may include an ADAS system 638. In some examples, the ADAS system 638 may include a SoC. The ADAS system 638 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keeping assist (LKA), blind spot warning (BSW), rear cross traffic warning (RCTW), collision warning system (CWS), lane centering (LC), and / or other features and functions.

[0180] The ACC system can utilize RADAR sensor 660, LIDAR sensor 664, and / or a camera. The ACC system can include longitudinal ACC and / or lateral ACC. Longitudinal ACC monitors and controls the distance to the vehicle immediately in front of vehicle 600, automatically adjusting the vehicle speed to maintain a safe distance from the vehicle in front. Lateral ACC maintains distance and, when necessary, recommends that vehicle 600 change lanes. Lateral ACC is related to other ADAS applications such as LCA and CWS.

[0181] CACC uses information from other vehicles, which can be received from other vehicles indirectly via a wireless link or through a network connection (e.g., through the Internet) via the network interface 624 and / or wireless antenna 626. A direct link can be provided by a vehicle-to-vehicle (V2V) communication link, while an indirect link can be an infrastructure-to-vehicle (I2V) communication link. Typically, the V2V communication concept provides information about the vehicle immediately ahead (e.g., the vehicle immediately ahead of the vehicle 600 and in the same lane as it), while the I2V communication concept provides information about traffic further ahead. The CACC system can include either or both of the I2V and V2V information sources. Given information about the vehicle ahead of the vehicle 600, CACC can be more reliable, and it has the potential to improve the smoothness of traffic flow and reduce road congestion.

[0182] The FCW system is designed to alert the driver to hazards so that the driver can take corrective action. The FCW system uses a front-facing camera and / or RADAR sensor 660 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating components. The FCW system can provide warnings in the form of, for example, audible, visual warnings, vibrations, and / or rapid brake pulses.

[0183] The AEB system detects an impending forward collision with another vehicle or other object and can automatically apply the brakes if the driver does not take corrective action within specified time or distance parameters. The AEB system can use a front-facing camera and / or RADAR sensor 660 coupled to a dedicated processor, DSP, FPGA and / or ASIC. When the AEB system detects a hazard, it typically first alerts the driver to take corrective action to avoid the collision, and if the driver does not take corrective action, the AEB system can automatically apply the brakes in an effort to prevent or at least mitigate the effects of the predicted collision. The AEB system can include technologies such as dynamic brake support and / or collision approach braking.

[0184] The LDW system provides visual, audible, and / or tactile warnings, such as steering wheel or seat vibrations, to alert the driver when the vehicle 600 crosses a lane marking. When the driver indicates an intention to leave the lane by activating a turn signal, the LDW system is deactivated. The LDW system may utilize a front-facing camera coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as a display, speaker, and / or vibration components.

[0185] The LKA system is a variation of the LDW system. If the vehicle 600 begins to leave the lane, the LKA system provides steering input or braking to correct the vehicle 600.

[0186] The BSW system detects and warns the driver of vehicles in the car's blind spot. The BSW system can provide visual, audible, and / or tactile alerts to indicate that merging or changing lanes is unsafe. The system can provide additional warnings when the driver uses a turn signal. The BSW system can use a rear-facing camera and / or RADAR sensor 660 coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as a display, speaker, and / or vibration component.

[0187] The RCTW system can provide visual, audible, and / or tactile notifications when an object is detected outside the range of the rear-mounted camera while the vehicle 600 is in reverse. Some RCTW systems include AEB to ensure that the vehicle brakes are applied to avoid a collision. The RCTW system can use one or more rear-mounted RADAR sensors 660 coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as a display, speaker, and / or vibration component.

[0188] Conventional ADAS systems may be prone to false positive results, which may be annoying and distracting to the driver, but are typically not catastrophic because the ADAS system alerts the driver and allows the driver to decide whether a safety condition actually exists and take action accordingly. However, in the autonomous vehicle 600, in the event of conflicting results, the vehicle 600 itself must decide whether to heed the results from the main computer or the auxiliary computer (e.g., the first controller 636 or the second controller 636). For example, in some embodiments, the ADAS system 638 can be a backup and / or auxiliary computer for providing perception information to the backup computer rationality module. The backup computer rationality monitor can run redundant and diverse software on hardware components to detect failures in perception and dynamic driving tasks. The output from the ADAS system 638 can be provided to the supervisory MCU. If the outputs from the main computer and the auxiliary computer conflict, the supervisory MCU must determine how to reconcile the conflict to ensure safe operation.

[0189] In some examples, the primary computer can be configured to provide a confidence score to the supervisory MCU, indicating the primary computer's confidence in the selected result. If the confidence score exceeds a threshold, the supervisory MCU can follow the primary computer's direction, regardless of whether the secondary computer provides conflicting or inconsistent results. In the event that the confidence score does not meet the threshold and the primary and secondary computers indicate different results (e.g., a conflict), the supervisory MCU can arbitrate between these computers to determine the appropriate result.

[0190] The supervisory MCU can be configured to run a neural network that is trained and configured to determine conditions under which the secondary computer provides a false alarm based, at least in part, on outputs from the primary computer and the secondary computer. Thus, the neural network in the supervisory MCU can learn when the output of the secondary computer can be trusted and when it cannot. For example, when the secondary computer is a RADAR-based FCW system, the neural network in the supervisory MCU can learn when the FCW system is identifying a metal object that is not actually a danger, such as a drain grate or manhole cover, which triggers an alarm. Similarly, when the secondary computer is a camera-based LDW system, the neural network in the supervisory MCU can learn to disregard the LDW when a cyclist or pedestrian is present and lane departure is actually the safest strategy. In embodiments that include a neural network running on the supervisory MCU, the supervisory MCU can include at least one of a DLA or a GPU suitable for running the neural network with associated memory. In preferred embodiments, the supervisory MCU can include and / or be included as a component of the SoC 604.

[0191] In other examples, the ADAS system 638 can include an auxiliary computer that uses traditional computer vision rules to perform ADAS functions. In this way, the auxiliary computer can use classic computer vision rules (if-then), and the presence of a neural network in the supervisory MCU can improve reliability, safety, and performance. For example, diverse implementations and intentional non-identity make the entire system more fault-tolerant, especially with respect to failures caused by software (or software-hardware interface) functions. For example, if there is a software vulnerability or bug in the software running on the main computer and the non-identical software code running on the auxiliary computer provides the same overall result, the supervisory MCU can be more confident that the overall result is correct and that the vulnerability in the software or hardware on the main computer did not cause a substantial error.

[0192] In some examples, the output of ADAS system 638 can be fed into the primary computer's perception block and / or the primary computer's dynamic driving task block. For example, if ADAS system 638 indicates a forward collision warning due to an object immediately ahead, the perception block can use this information when identifying the object. In other examples, the secondary computer can have its own neural network that is trained and thus reduces the risk of false positives as described herein.

[0193] The vehicle 600 may also include an infotainment SoC 630 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system may not be an SoC and may include two or more separate components. The infotainment SoC 630 may include a combination of hardware and software that can be used to provide audio (e.g., music, personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), phone (e.g., hands-free calling), network connectivity (e.g., LTE, WiFi, etc.), and / or information services (e.g., a navigation system, rear parking assistance, radio data system, vehicle-related information such as fuel level, total distance covered, brake fuel level, oil level, door open / closed, air filter information, etc.) to the vehicle 600. For example, the infotainment SoC 630 may include a radio, a disc player, a navigation system, a video player, USB and Bluetooth connectivity, an onboard computer, in-vehicle entertainment, WiFi, steering wheel audio controls, hands-free voice controls, a head-up display (HUD), an HMI display 634, a telematics device, a control panel (e.g., for controlling and / or interacting with various components, features, and / or systems), and / or other components. The infotainment SoC 630 may further be used to provide information (e.g., visual and / or auditory) to a user of the vehicle, such as information from an ADAS system 638, autonomous driving information such as planned vehicle maneuvers, trajectories, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0194] The infotainment SoC 630 may include GPU functionality. The infotainment SoC 630 may communicate with other devices, systems, and / or components of the vehicle 600 via a bus 602 (e.g., a CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 630 may be coupled to a supervisory MCU so that in the event of a failure of a primary controller 636 (e.g., a primary and / or backup computer of the vehicle 600), the infotainment system's GPU may perform some self-driving functions. In such an example, the infotainment SoC 630 may place the vehicle 600 in a driver-safe parking mode as described herein.

[0195] The vehicle 600 may also include an instrument cluster 632 (e.g., a digital instrument panel, an electronic instrument cluster, a digital instrument panel, etc.). The instrument cluster 632 may include a controller and / or a supercomputer (e.g., a separate controller or a supercomputer). The instrument cluster 632 may include a set of instruments, such as a speedometer, fuel level, oil pressure, a tachometer, an odometer, a turn indicator, a shift position indicator, a seat belt warning light, a parking brake warning light, an engine check light, airbag (SRS) system information, lighting controls, safety system controls, navigation information, etc. In some examples, information may be displayed and / or shared between the infotainment SoC 630 and the instrument cluster 632. In other words, the instrument cluster 632 may be included as part of the infotainment SoC 630, or vice versa.

[0196] Figure 6D For cloud-based servers and Figure 6A 6. System diagram of communication between an example autonomous vehicle 600. System 676 may include a server 678, a network 690, and a vehicle including vehicle 600. Server 678 may include multiple GPUs 684(A)-684(H) (collectively referred to herein as GPUs 684), PCIe switches 682(A)-682(H) (collectively referred to herein as PCIe switches 682), and / or CPUs 680(A)-680(B) (collectively referred to herein as CPUs 680). GPUs 684, CPUs 680, and PCIe switches may be interconnected with a high-speed interconnect and / or PCIe connection 686, such as, for example and without limitation, an NVLink interface 688 developed by NVIDIA. In some examples, GPUs 684 are connected via NVLink and / or NVSwitch SoCs, and GPUs 684 and PCIe switches 682 are connected via a PCIe interconnect. Although eight GPUs 684, two CPUs 680, and two PCIe switches are illustrated, this is not intended to be limiting. Depending on the embodiment, each of the servers 678 can include any number of GPUs 684, CPUs 680, and / or PCIe switches. For example, each of the servers 678 can include eight, sixteen, thirty-two, and / or more GPUs 684.

[0197] Server 678 can receive image data from a vehicle via network 690, the image data representing images showing unexpected or changed road conditions, such as recently begun road construction. Server 678 can transmit neural network 692, updated neural network 692, and / or map information 694, including information about traffic and road conditions, via network 690 and to the vehicle. Updates to map information 694 can include updates to HD map 622, such as information about construction sites, potholes, curves, flooding, or other obstacles. In some examples, neural network 692, updated neural network 692, and / or map information 694 can be generated from new training and / or data received from any number of vehicles in the environment and / or based on experience with training performed at a data center (e.g., using server 678 and / or other servers).

[0198] Server 678 can be used to train a machine learning model (e.g., a neural network) based on training data. The training data can be generated by the vehicle and / or can be generated in simulation (e.g., using a game engine). In some examples, the training data is labeled (e.g., where the neural network benefits from supervised learning) and / or undergoes other preprocessing, while in other examples, the training data is not labeled and / or preprocessed (e.g., where the neural network does not require supervised learning). Once the machine learning model is trained, the machine learning model can be used by the vehicle (e.g., transmitted to the vehicle via network 690), and / or the machine learning model can be used by server 678 to remotely monitor the vehicle.

[0199] In some examples, server 678 can receive data from the vehicle and apply the data to the latest real-time neural network for real-time intelligent reasoning. Server 678 can include a deep learning supercomputer and / or a dedicated AI computer powered by GPU 684, such as the DGX and DGX Station machines developed by NVIDIA. However, in some examples, server 678 can include the deep learning infrastructure of a data center using only CPU power.

[0200] The deep learning infrastructure of server 678 may be capable of rapid real-time inference and may use this capability to assess and verify the health of the processors, software, and / or associated hardware in vehicle 600. For example, the deep learning infrastructure may receive periodic updates from vehicle 600, such as an image sequence and / or objects that vehicle 600 has located in the image sequence (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure may run its own neural network to identify objects and compare them to the objects identified by vehicle 600, and if the results do not match and the infrastructure concludes that the AI ​​in vehicle 600 has malfunctioned, server 678 may transmit a signal to vehicle 600 instructing the vehicle's 600 fail-safe computer to take control, notify passengers, and complete a safe parking maneuver.

[0201] For inference, the server 678 may include a GPU 684 and one or more programmable inference accelerators (e.g., NVIDIA's TensorRT). The combination of GPU-powered servers and inference acceleration can enable real-time responses. In other examples, such as where performance is less important, CPU, FPGA, and other processor-powered servers can be used for inference.

[0202] Example computing device

[0203] Figure 7 FIG2 is a block diagram of an example computing device 700 suitable for implementing some embodiments of the present disclosure. The computing device 700 may include a bus 702 that directly or indirectly couples the following devices: memory 704, one or more central processing units (CPUs) 706, one or more graphics processing units (GPUs) 708, a communication interface 710, input / output (I / O) ports 712, input / output components 714, a power supply 716, and one or more presentation components 718 (e.g., a display).

[0204] although Figure 7 The various blocks of are shown as being connected via a bus 702 having lines, but this is not intended to be limiting and is provided for clarity only. For example, in some embodiments, presentation components 718 such as a display device may be considered I / O components 714 (e.g., if the display is a touch screen). As another example, CPU 706 and / or GPU 708 may include memory (e.g., memory 704 may represent a storage device in addition to the memory of GPU 708, CPU 706, and / or other components). In other words, Figure 7The term computing device is illustrative only. No distinction is made between categories such as "workstation," "server," "laptop," "desktop," "tablet," "client device," "mobile device," "handheld device," "game console," "electronic control unit (ECU)," "virtual reality system," and / or other device or system types, as all are considered within the Figure 7 within the range of computing devices.

[0205] The bus 702 may represent one or more buses, such as an address bus, a data bus, a control bus, or a combination thereof. The bus 702 may include one or more bus types, such as an Industry Standard Architecture (ISA) bus, an Extended Industry Standard Architecture (EISA) bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe) bus, and / or another type of bus.

[0206] Memory 704 may include any of a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by computing device 700. Computer-readable media may include volatile and non-volatile media and removable and non-removable media. By way of example and not limitation, computer-readable media may include computer storage media and communication media.

[0207] Computer storage media may include volatile and non-volatile media and / or removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 704 may store computer-readable instructions (e.g., representing programs and / or program elements, such as an operating system). Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage devices, magnetic cassettes, magnetic tape, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by the computing device 700. As used herein, computer storage media does not include signals themselves.

[0208] Communication media may embody computer-readable instructions, data structures, program modules, and / or other data types in a modulated data signal such as a carrier wave or other transport mechanism, and include any information transport media. The term "modulated data signal" may refer to a signal that has one or more of its characteristics set or changed in such a way as to encode information into the signal. By way of example and not limitation, communication media may include wired media such as a wired network or a direct wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Any combination of the above should also be included within the scope of computer-readable media.

[0209] The CPU 706 can be configured to execute computer-readable instructions to control one or more components of the computing device 700 to perform one or more of the methods and / or processes described herein. Each of the CPUs 706 may include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of processing a large number of software threads simultaneously. The CPU 706 may include any type of processor and may include different types of processors, depending on the type of computing device 700 implemented (e.g., a processor with fewer cores for mobile devices and a processor with more cores for servers). For example, depending on the type of computing device 700, the processor may be an ARM processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplementary coprocessors such as math coprocessors, the computing device 700 may also include one or more CPUs 706.

[0210] The GPU 708 can be used by the computing device 700 to render graphics (e.g., 3D graphics). The GPU 708 can include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. The GPU 708 can generate pixel data for outputting an image in response to a rendering command (e.g., a rendering command from the CPU 706 received via a host interface). The GPU 708 can include graphics memory such as display memory for storing pixel data. The display memory can be included as part of the memory 704. The GPU 708 can include two or more GPUs operating in parallel (e.g., via a link). When combined, each GPU 708 can generate pixel data for different parts of the output image or for different output images (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU can include its own memory or can share memory with other GPUs.

[0211] In examples where computing device 700 does not include GPU 708 , CPU 706 may be used to render graphics.

[0212] The communication interface 710 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 700 to communicate with other computing devices via an electronic communication network, including wired and / or wireless communications. The communication interface 710 may include components and functionality that enable communication over any of a number of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communicating over Ethernet), low-power wide-area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet.

[0213] The I / O ports 712 can enable the computing device 700 to be logically coupled to other devices including I / O components 714, presentation components 718, and / or other components, some of which can be built into (e.g., integrated into) the computing device 700. Illustrative I / O components 714 include a microphone, a mouse, a keyboard, a joystick, a game pad, a game controller, a satellite dish, a scanner, a printer, a wireless device, and the like. The I / O components 714 can provide a natural user interface (NUI) that processes user-generated mid-air gestures, voice, or other physiological input. In some examples, the input can be transmitted to an appropriate network element for further processing. The NUI can implement any combination of voice recognition, stylus recognition, facial recognition, biometric recognition, gesture recognition on and adjacent to the screen, mid-air gestures, head and eye tracking, and touch recognition associated with the display of the computing device 700 (as described in more detail below). The computing device 700 can include a depth camera such as a stereo camera system, an infrared camera system, an RGB camera system, touch screen technology, and combinations thereof for gesture detection and recognition. Additionally, computing device 700 may include an accelerometer or gyroscope to enable motion detection (e.g., as part of an inertial measurement unit (IMU)). In some examples, the output of the accelerometer or gyroscope may be used by computing device 700 to render immersive augmented or virtual reality.

[0214] The power supply 716 may include a hardwired power supply, a battery power supply, or a combination thereof. The power supply 716 may provide power to the computing device 700 to enable the components of the computing device 700 to operate.

[0215] The presentation component 718 may include a display (e.g., a monitor, a touch screen, a television screen, a head-up display (HUD), other display types, or a combination thereof), speakers, and / or other presentation components. The presentation component 718 may receive data from other components (e.g., the GPU 708, the CPU 706, etc.) and output the data (e.g., as images, video, sound, etc.).

[0216] The present disclosure can be described in the general context of machine-usable instructions or computer code executed by a computer or other machine such as a personal digital assistant or other handheld device, including computer-executable instructions such as program modules. Generally, program modules including routines, programs, objects, components, data structures, etc. refer to code that performs a specific task or implements a specific abstract data type. The present disclosure can be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc. The present disclosure can also be practiced in a distributed computing environment where tasks are performed by remote processing devices linked through a communication network.

[0217] As used herein, the statement "and / or" with respect to two or more elements should be interpreted as referring to only one element or combination of elements. For example, "element A, element B and / or element C" may include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B and C. In addition, "at least one of element A or element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Further, "at least one of element A and element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0218] The subject matter of the present disclosure is described in detail herein to meet statutory requirements. However, the description itself is not intended to limit the scope of the present disclosure. On the contrary, the inventors have contemplated that the claimed subject matter may also be embodied in other ways to include steps that are different from the steps described herein in conjunction with other current or future technologies, or combinations of similar steps. Moreover, although the terms "step" and / or "block" may be used herein to imply different elements of the method employed, these terms should not be interpreted as implying any particular order among or between the various steps disclosed herein, unless the order of the steps is explicitly described.

Claims

1. A method for future trajectory prediction, comprising: receiving sensor data generated by one or more sensors of a vehicle within an environment over a period of time; determining at least one past location of the active person during the time period based at least in part on the sensor data; generating a first image representing the at least one past location of the active person; receiving first data representing map information corresponding to the environment; generating a second image representing the map information; applying image data representing the first image and the second image to a deep neural network (DNN); calculating, using the DNN and based at least in part on the image data, second data corresponding to a vector field and third data corresponding to a confidence field, wherein at least one point within the vector field corresponds to a motion vector representing a past position of the agent at a time before the future time and after the time period; as well as At least one future location of the agent at a future time after the time period is determined based at least in part on the vector field and the confidence field.

2. The method of claim 1 , wherein generating the first image comprises: Rasterizing the past position and generating the second image includes rasterizing the map information. 3 . The method of claim 1 , wherein the map information is oriented relative to the location of the vehicle and the orientation of the vehicle at a most recent time within the time period. 4 . The method of claim 1 , wherein the first image is oriented with the vehicle at its center, and the at least one past position of the active person is oriented within the first image relative to the vehicle's position. The method of claim 1 , wherein the first image and the second image are rasterized in a top view.

6. The method of claim 1, wherein at least one point within the confidence field corresponds to a confidence that the point corresponds to the at least one future location of the agent.

7. The method of claim 1 , wherein the neural network comprises a recurrent neural network (RNN), and further wherein: The RNN includes at least a first encoder-decoder architecture that receives as input the one or more past positions of the agent; And the RNN includes a second encoder-decoder architecture that receives as input one or more predicted future positions of the agent, the one or more predicted future positions of the agent being output from the first encoder-decoder architecture of a third encoder-decoder architecture of the RNN.

8. The method of claim 7, wherein state information calculated using the first encoder-decoder architecture is shared with the second encoder-decoder architecture. 9 . The method of claim 1 , wherein the map information includes a location of at least one of a lane, an exit, a merge, or an intersection. 10 . The method of claim 9 , wherein the map information further includes at least one of a free space boundary location or wait condition information.

11. The method according to claim 1 , wherein: The second data is calculated using a first layer flow of the DNN; and the third data is calculated using a second layer flow of the DNN that is different from the first layer flow.

12. The method of claim 11, wherein the DNN is trained to calculate the second data using a first loss function, and to calculate the third data using a second loss function different from the first loss function.

13. The method of claim 1, wherein a loss function is used to train the DNN factors in the weights of each instance based at least in part on the relevance of the instances of training data.

14. A method according to claim 13, wherein the correlation is determined at least in part based on at least one of: a standard deviation of the speed of at least one activity in the instances of the training data; a ratio of the standard deviation of the speed of at least one activity in the longitudinal direction and the lateral direction in the instances of the training data; or a change in the standard deviation of the speed of at least one activity in the instances of the training data.

15. The method of claim 1 , wherein the at least one future location is determined using a cluster, the cluster comprising: determining a plurality of points within the confidence field that are above a threshold confidence value; determining a centroid of the plurality of points; and determining the at least one future position using the centroid.

16. A method according to claim 1, wherein the at least one future position is determined by: determining a plurality of points in the confidence field corresponding to the future time based at least in part on a plurality of motion vectors within the vector field corresponding to another future time after the future time; calculating a weighted average of confidence values ​​from the confidence field corresponding to the plurality of points; and using the weighted average to determine the at least one future position corresponding to the future time.

17. The method according to claim 1, further comprising: Identifying a validated training data instance from a set of training data by: analyzing the set of training data using at least one of another deep neural network (DNN) trained for data mining, heuristics, or statistical models; determining a training data instance based at least in part on the analyzing; and validating the training data instance to generate the validated training data instance.

18. The method according to claim 17, further comprising: The DNN is trained using the validated training data instances and the additional training data instances, the training including weighting the validated training data instances differently than the additional training data instances.

19. A method for future trajectory prediction, comprising: calculating, using a neural network, first data representing a first confidence map corresponding to a first future time, second data representing a second confidence map corresponding to a second future time subsequent to the first future time, and third data representing a vector field corresponding to the second future time; determining one or more second points on the second confidence map indicating a second future location of the agent; determining one or more vectors from the vector field corresponding to the one or more second points, the one or more vectors each representing a predicted position of the agent in the first confidence map; determining one or more first points on the first confidence map based at least in part on the one or more vectors; as well as A first future location of the agent prior to the second future location is determined based at least in part on the one or more first points.

20. The method of claim 19, wherein calculating the first data uses a first head of the neural network, and calculating the second data uses a second head of the neural network.

21. The method of claim 19, wherein the map information and at least one previous position are oriented from a top-down perspective and relative to the position of the ego vehicle before being applied to the neural network.

22. The method of claim 19, wherein the neural network is an encoder-decoder recurrent neural network (RNN).

23. The method of claim 19, wherein determining the first future location comprises: Clustering is performed on the values ​​corresponding to the one or more first points.

24. The method of claim 19, wherein determining the first future location comprises: A weighted average of the values ​​corresponding to the one or more first points is determined.

25. The method of claim 19, wherein calculating the first data and calculating the second data are based at least in part on third data representing map information and fourth data representing a previous position of an agent applied to the neural network, the third data and the fourth data being applied to the neural network as image data.

26. The method of claim 25, wherein calculating the first data and calculating the second data are further based at least in part on fifth data representing state information of an agent applied to the neural network.

27. The method of claim 19, further comprising: A predicted future trajectory is generated for the agent based at least in part on the first future location and the second future location.

28. A system for future trajectory prediction, comprising: one or more sensors of the vehicle for generating sensor data; A computing device comprising one or more processing devices and one or more memory devices communicatively coupled to the one or more processing devices, the one or more memory devices storing programming instructions on the one or more processing devices that, when executed by the processor, cause the following to be instantiated: an image generator for generating a first image representing map information and a second image representing one or more previous locations of an agent; a neural network executor for calculating, using a neural network and based at least in part on the first image and the second image: a first confidence map corresponding to a first future time; a second confidence map corresponding to a second future time after the first future time; and a vector field corresponding to the second future time; a future position determiner configured to: determine one or more points on the second confidence map indicating a second future position of the agent; determining one or more vectors corresponding to the one or more points from the vector field; and determining a first future location of the agent that is prior to the second future location based at least in part on the one or more vectors and the first confidence map; and a trajectory generator for generating a trajectory of the agent using the first future position and the second future position.

Images