Single and cross-sensor object tracking using feature descriptor mapping in autonomous systems and applications

Abstract

The present disclosure relates to single and cross-sensor object tracking using feature descriptor mapping in autonomous systems and applications. In various examples, real-time perception from vehicle sensors can be leveraged to generate object tracking paths for a vehicle to facilitate real-time or near real-time navigation control. For example, a deep neural network (DNN) can be trained to compute various outputs, such as a feature descriptor map that includes feature descriptor vectors corresponding to objects contained in one or more sensor fields of view. The outputs can be decoded and / or otherwise post-processed to reconstruct object tracks and determine suggested or potential paths for navigating the vehicle.

Description

BACKGROUND

[0001] Autonomous driving systems and advanced driver assistance systems (ADAS) can use sensors such as cameras, LIDAR (light detection and ranging) sensors, RADAR (radio detection and ranging) sensors, and / or the like to perform various tasks such as object detection, object tracking, lane keeping, lane changing, lane assignment, camera calibration, turning, path planning, and / or localization. For example, to enable autonomous and ADAS systems to function independently and effectively, an understanding of the environment around the machine must be achieved. This understanding can include information about object locations and object paths or trajectories, and the machine can use this information when making path planning or control decisions such as what path or trajectory to follow.

[0002] For example, information about object locations and object paths in the environment of an autonomous or semi-autonomous machine is important when planning a machine path, avoiding objects, and / or making control decisions such as where to stop, what path to use to safely cross an intersection, where other vehicles or pedestrians can be located, and the like.

[0003] Conventional object tracking systems use motion models or convolutional DNN-based models, both of which heavily rely on object detectors. For example, some motion models use key points within a bounding shape such as scale-invariant feature transform (SIFT) algorithms and / or Kanade-Lucas-Tomasi (KLT) feature trackers as feature descriptors. Motion model approaches attempt to predict a current bounding shape location using previous key point locations and a pre-defined motion model, and tracking is achieved by computing an intersection over union (IOU) score between the bounding shape location predicted by the motion model and the bounding shape location predicted by the object detector. However, simple key points often fail to capture high-level semantic features of individual objects. In other words, object tracking using motion models is highly dependent on the continuity and visibility of key points. Thus, changes in camera viewpoints and partial observations often cause tracking to fail.

[0004] Similarly, motion models fail when objects move in a way that does not follow a pre-defined motion model - for example, tracking errors often occur when objects are close to each other or cross each other in opposite directions. For example, when a group of people cross a street in opposite directions, a motion model can fail to accurately track the path of each person because some actors occlude other actors. These motion models often heavily rely on manually adjusted thresholds such as a minimum number of key points and a confidence threshold, which need to be re-adjusted for different road conditions and scenarios.

[0005] Recent approaches that rely on convolutional DNN-based models leverage the feature encoding capabilities of DNNs to track objects. Typically, these DNN-based models are first trained as part of an object detection pipeline and then post-detection tracking is performed using commercially available matching or association algorithms. However, neither the DNN features nor the association algorithms are optimized for object tracking, which often results in object identity switches - e.g., when two objects cross paths, the tracking can mistake each actor for the other. Moreover, the computational cost of extracting and caching DNN features for each detected object is high, and therefore does not scale well to tracking a large number of objects. SUMMARY

[0006] Embodiments of the present disclosure relate to object tracking using learned feature vectors for single-sensor applications and multi-sensor applications. Systems and methods are disclosed that leverage outputs from various sensors of a machine to track objects in a view of a single sensor or across multiple sensors using vector-based feature descriptors. Further, systems and methods are disclosed that leverage outputs from various sensors to train an artificial intelligence engine using vector triplets. For example, a DNN can be directly trained for object tracking using pixel-level feature descriptors as compared to conventional systems such as those described above. Accordingly, a DNN-based model can track objects within a single sensor - or across multiple sensors at the same time instance - even if the objects are only partially observed, occluded, cross paths, etc. Object tracking data from single-sensor tracking and / or cross-sensor tracking can be used by one or more downstream components of the ego machine - such as a world model manager, a path planner, a control component, a localization component, an obstacle avoidance component, an actuation component, etc. - to perform one or more operations for controlling the ego machine through an environment. BRIEF DESCRIPTION OF DRAWINGS

[0007] The present systems and methods for learned feature vectors for single and cross-sensor object tracking are described in detail below with reference to the accompanying drawings, wherein:

[0008] Figure 1 depicted is a dataflow diagram showing an example training process for training a machine learning model to compute feature descriptor maps, in accordance with some embodiments of the present disclosure;

[0009] Figure 2A depicted is an example visualization of sensor data instances, in accordance with some embodiments of the present disclosure;

[0010] Figure 2B depicted is an example visualization of annotations for generating ground truth data corresponding to instances of sensor data, in accordance with some embodiments of the present disclosure;

[0011] Figures 2C-2Ddepicts an example feature descriptor vector graph corresponding to training data for a neural network according to some embodiments of the present disclosure;

[0012] Figures 3A-3B depicts example positive samples, negative samples, and anchor samples from a feature descriptor vector map according to some embodiments of the present disclosure;

[0013] Figures 4A-4B depicts a flowchart of an example method for training a machine learning model to track an object according to some embodiments of the present disclosure;

[0014] Figure 5 depicts a data flow diagram illustrating an example process for tracking an object using a machine learning model according to some embodiments of the present disclosure;

[0015] Figures 6A-6C depicts an example feature descriptor graph according to some embodiments of the present disclosure;

[0016] Figure 7 depicts a flowchart of an example method for tracking an object using a machine learning model according to some embodiments of the present disclosure;

[0017] Figure 8A is an illustration of an example autonomous vehicle according to some embodiments of the present disclosure;

[0018] Figure 8B According to some embodiments of the present disclosure Figure 8A Examples of camera positions and fields of view for autonomous vehicles;

[0019] Figure 8C According to some embodiments of the present disclosure Figure 8A a block diagram of an example system architecture for an example autonomous vehicle;

[0020] Figure 8D According to some embodiments of the present disclosure, a method for Figure 8A System diagram of an example of communication between autonomous vehicles;

[0021] Figure 9 is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure; and

[0022] Figure 10 is a block diagram of an example data center suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0023] Systems and methods related to learning feature vectors for single and cross-sensor object tracking are disclosed. Although the present disclosure can be described with respect to an example autonomous vehicle 800 (alternatively referred to herein as "vehicle 800" or "ego vehicle 800," examples of which are described with respect to Figures 8A-8D described, this is not intended to be limiting. For example, the systems and methods described herein can be used by non-limiting examples of non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more advanced driver assistance systems (ADAS)), manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles connected to one or more trailers, airships, boats, shuttles, emergency vehicles, motorcycles, electric or motorized bicycles, airplanes, engineering vehicles, underwater vehicles, drones, and / or other vehicle types. Furthermore, although the present disclosure can be described with respect to tracking objects across multiple sensors and / or over time, this is not intended to be limiting, and the systems and methods described herein can be used for augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications, and / or any other technical space that can use tracking objects across multiple sensors and / or over time.

[0024] As described herein, and in contrast to conventional approaches to object tracking, the present systems and methods provide techniques for tracking objects in real-time or near real-time using output from sensors (e.g., cameras, RADAR, sensors, LIDAR, etc.) of a machine. To accomplish this, the system trains a DNN using ground truth vector triplets. Generally, training data is provided as input to the DNN, and the DNN uses the training data to generate a descriptor map that includes at least one-dimensional (ID) vector feature descriptors of pixels associated with an object. The training data can be selected to include multi-vector triplets that include an anchor vector (e.g., associated with a first pixel known to correspond to an object), a positive vector (e.g., associated with a second pixel known to correspond to the object), and a negative vector (e.g., associated with a third pixel known to correspond to a different object than the object). In this way, when the DNN computes an output feature vector using the training data, these known anchor, positive, and negative vectors can be used as ground truth to train the DNN to minimize the distance between the positive and anchor vectors while maximizing the distance between the anchor and negative vectors. In this way, the DNN can be trained to compute similar or identical feature vector descriptors for the same object and different feature vectors for different objects.

[0025] In particular, the ground truth vector triplets can be selected using an object detection algorithm that labels objects in an image (or other sensor data representation of a space) with a bounding shape. In some aspects, tracks shorter than a predetermined minimum track length are removed. The predetermined minimum track length can vary based on the object class (e.g., vehicle or pedestrian). Further, in some aspects, the last 30 frames of a data sequence (e.g., a video) can be filtered to enhance the accuracy of the ground truth data by eliminating false object detection errors. In some aspects, to reduce the training burden, a data sequence longer than a predetermined threshold can be sampled every n frames to reduce the data sequence length. A track ID can then be assigned to each bounding shape - for example, the same track ID value can be assigned to each pixel within the same bounding shape, and a value of zero (0) can be assigned to background pixels (e.g., pixels not covered by any bounding shape). During training, all feature vectors within a particular bounding shape (e.g., labeled with the same track ID) are considered positive samples, while all feature vectors belonging to other objects are considered negative samples.

[0026] Vector triplets can be created from feature descriptor maps across space and / or time. For example, in a single sensor tracking scenario, at least two descriptor maps (t 1-k descriptors at t Figure 1 and descriptor map 2 at t1may be used to create vector triplets. The positive sample can be one or more vectors from the positive sample of descriptor Figure 1 map 2, while the negative sample can be one or more vectors from the negative sample of descriptor Figure 1 map 2 (e.g., a vector within another bounding shape). The anchor can be one or more vectors from the positive sample of descriptor map 2.

[0027] Similarly, vector triplets can be generated to train a DNN for cross-sensor tracking - for example, at the same time step. In such examples, the positive sample can be one or more vectors from the positive sample of a first sensor descriptor map (e.g., a vector within a bounding shape with the same track ID), while the negative sample can be one or more vectors from the negative sample of the first sensor descriptor map (e.g., a vector within another bounding shape). The anchor can be one or more vectors from the positive sample of a second sensor descriptor map.

[0028] For a given triple, a triple loss function can be used to minimize the distance between the anchor feature descriptor and the positive feature descriptor (e.g., distance "d" between anchor "a" and positive "p"). In other words, the triple loss minimizes d(a, p). Similarly, the triple loss function can minimize the distance between the anchor and the negative (e.g., minimize d(a, n)). Thus, this loss function can cause the DNN to learn useful feature descriptors in the embedding space such that features of the same object across space and / or time are closer, while features of different objects are farther apart. To prevent the DNN from learning trivial (all-zero) feature vectors, a margin value can be added to the maximum margin loss. Moreover, the triple loss can be computed at the pixel level even if the ground truth data includes partially ambiguous objects, to cause the DNN to learn useful feature descriptors.

[0029] Advantageously, the use of vector triples facilitates DNN training using multiple mining options. Generally, there are three types of triples: simple triples, hard triples, and semi-hard triples. In one or more embodiments, simple triples have a triple loss of 0, hard triples are triples in which the negative sample is closer to the anchor than the positive sample, and semi-hard triples are triples in which the negative sample is not closer to the anchor than the positive sample, but still has a positive loss. In embodiments, to speed up training and increase the discriminative ability of the DNN, only semi-hard and hard triples can be used during training. However, during training in certain aspects, simple triples, semi-hard triples, and hard triples can be used. Moreover, using vector triples facilitates epoch or batch mining. For example, the vector sum and boundary shapes of the entire training set can be computed before training the DNN on an epoch-long triple. Alternatively, a batch-all or batch-hard approach to triple mining can be implemented. For batch-all triple mining, all valid triples are selected for a given batch, the loss is averaged over the hard and semi-hard triples, and the simple triples are ignored. For batch-hard triple mining, the hardest positive and hardest negative in the batch for each anchor are selected, and only the triple loss function for the hardest triples is computed.

[0030] In embodiments, training the DNN to track objects using vector triplets facilitates deployment integration with minimal memory burden. For example, by using pixel-level to bounding shape-level conversion, the trained or deployed DNN can be integrated into legacy ego machine systems with minimal downstream impact. Illustratively, during deployment, pixel-level feature descriptors can be converted to bounding shape-level feature descriptors by averaging all pixel-level feature descriptors within a particular bounding shape. The average can then be assigned as the bounding shape-level feature descriptor, and the bounding shape-level feature descriptor can be used to track the corresponding object across frames (e.g., from the same sensor or across different sensors).

[0031] A relatively straightforward post-processing step can facilitate single-sensor and multi-sensor object tracking. For applications in single-sensor tracking, the DNN processes consecutive frames (e.g., tx, tx-1, tx-2, tx-3, etc.) respectively and generates bounding shape-level feature descriptors for detected objects as described above. Object association scores between each bounding shape feature descriptor can be computed, and differences in feature descriptors can be computed to identify the same object across frames. For example, where the distance between feature descriptors is less than a threshold, the bounding shapes in different frames can be determined to correspond to the same object. In some aspects, the object association score is a cosine distance between each bounding shape feature descriptor. In embodiments, for tracking purposes, a KNN algorithm can be used to find the best match.

[0032] Reference Figure 1 , Figure 1 is an example data flow illustrating an example process 100 for training a DNN to track objects using output of one or more sensors of a machine, in accordance with some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are set forth only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, groupings of functions, etc.) can be used in addition to or instead of those shown, and some elements can be omitted altogether. Further, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in conjunction with other components, and in any suitable combination and location. Various functions described herein as being performed by entities can be stored in and executed by hardware, firmware and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. In some embodiments, the systems, methods, and processes described herein can be implemented using those similar to the components, features, and / or functions of Figures 8A-8D example autonomous vehicle 800 of FIG. 1, Figure 9 example computing device 900 of FIG. 2, and / or Figure 10 example data center 1000 of FIG. 3.

[0033] At a high level, the process 100 can include one or more machine learning models 104 that receive one or more inputs, such as sensor data 102, and generate one or more outputs, such as outputs (e.g., one or more feature descriptor maps 106). As used with reference to training, the sensor data 102 can be referred to as training data. Although the sensor data 102 is primarily discussed with respect to image data representing images, this is not intended to be limiting, and the sensor data 102 can include other types of sensor data for object detection and / or object tracking, such as LIDAR data, SONAR data, RADAR data, etc. - e.g., as produced by one or more sensors of the vehicle 800 described with reference to Figures 8A-8D

[0034] The process 100 can include generating and / or receiving sensor data 102 from one or more sensors. As a non-limiting example, the sensor data 102 can be received from one or more sensors of a machine (e.g., the vehicle 800 of Figures 8A-8D The sensor data 102 can be used by the ego machine 800, and within the process 100, for tracking objects across frames and / or across sensors. The sensor data 102 can include, but is not limited to, sensor data 102 from any sensor of a vehicle, including, for example, and with reference to Figures 8A-8C ​one or more RADAR sensors 860, one or more ultrasonic sensors 862, one or more LIDAR sensors 864, one or more inertial measurement unit (IMU) sensors 866 (e.g., one or more accelerometers, one or more gyroscopes, one or more magnetometers, one or more magnetorquers, etc.), one or more microphones 876, one or more stereo cameras 868, one or more wide-angle cameras 870 (e.g., fisheye cameras), one or more infrared cameras 872, one or more surround cameras 874 (e.g., 360-degree cameras), one or more long-range and / or mid-range cameras 878, one or more speed sensors 844 (e.g., to measure a speed of the vehicle 800), and / or other sensor types. As another example, the sensor data 102 can include virtual sensor data generated from any number of sensors of a virtual vehicle or other virtual object. In such examples, the virtual sensors can correspond to a virtual vehicle or other virtual object in a simulated environment (e.g., to test, train, and / or validate neural network performance), and the virtual sensor data can represent sensor data captured by the virtual sensors within the simulated or virtual environment. Thus, by using virtual sensor data, one or more machine learning models 104 described herein can be tested, trained, and / or validated using simulated data in a simulated environment, which can allow for testing of more extreme scenarios outside of real-world environments where such testing can be less safe.

[0035] In some embodiments, the sensor data 102 can include image data representing one or more images, image data representing a video (e.g., a snapshot of a video), and / or sensor data representing a representation of a sensor’s sensory field (e.g., a depth map for a LIDAR sensor, a value map for an ultrasonic sensor, etc.). Where the sensor data 102 includes image data, any type of image data format can be used, such as but not limited to compressed images (e.g., Joint Photographic Experts Group (JPEG) or luminance / chrominance (YUV) formats), as compressed images sourced from frames of a compressed video format (e.g., H.264 / Advanced Video Coding (AVC) or H.265 / High Efficiency Video Coding (HEVC)), such as raw images sourced from a Red Clear Blue (RCCB), Red Clear Clear (RCCC), or other type of imaging sensor, and / or other formats. Additionally, in some examples, the sensor data 102 can be used within the process 100 without any pre-processing (e.g., in raw or captured format), while in other examples, the sensor data 102 can undergo pre-processing (e.g., noise balancing, demosaicing, scaling, cropping, enhancement, white balancing, tone curve adjustment, etc., such as using a sensor data pre-processor (not shown)). As used herein, the sensor data 102 can refer to unprocessed sensor data, pre-processed sensor data, or a combination thereof.

[0036] The sensor data 102 used for training can include raw images (e.g., as captured by one or more image sensors), down-sampled images, up-sampled images, cropped or region of interest (ROI) images, otherwise enhanced images, and / or combinations thereof. The images (and / or other sensor data 102) and corresponding ground truth data (e.g., one or more trajectory ID maps 118) can be used to train one or more machine learning models 104. The ground truth data (e.g., one or more trajectory ID maps 118) can include annotations, labels, masks, maps, etc. For example, in some embodiments, the ground truth data can include a trajectory map. The encoder 116 can encode the ground truth data (e.g., a trajectory ID map 120) using object detection 108 and annotations. In some embodiments, the object detection 108 can include one or more bounding shapes 110 and one or more trajectory ID labels 114.

[0037] With reference to object detections 108 and annotations 112, in some examples, object detections 108 and annotations 112 can be generated within a drawing program (e.g., an annotation program), a computer-aided design (CAD) program, a labeling program, another type of program suitable for generating annotations 112 and / or other ground truth data (e.g., one or more track ID maps 118), and / or can be hand-drawn. In any example, object detections 108, annotations 112, and / or ground truth data (e.g., one or more track ID maps 118) can be synthetically generated (e.g., generated from a computer model or rendering), realistically generated (e.g., designed and generated from real-world data), machine-automated (e.g., using object detection and tracking to extract features from the data and subsequently generate labels), manually annotated (e.g., a human annotator or annotation expert defines the location of the labels), and / or a combination thereof. Object detection 108 can include labeling the object with a bounding shape 110 corresponding to the object of interest in the sensor data 102. In some examples, the bounding shape can be defined by a set of line segments surrounding pixels corresponding to the object in the sensor data 102. The line segments may be straight and / or curved. One or more trajectory ID tags 114 may be values ​​that facilitate identification between any tagged item and any other tagged item. For example, a trajectory ID tag 114 may be generated for each object, boundary shape, and / or pixel within a boundary shape included in the sensor data. In some embodiments, a trajectory ID tag for a particular object is assigned to a boundary shape corresponding to the object. Similarly, in some embodiments, a trajectory ID tag for a particular boundary shape may be assigned to each pixel within the boundary shape. Additionally, a default or empty trajectory ID tag may define an area of ​​sensor data that is not associated with an object. For example, background features in the sensor data may be labeled with a trajectory ID tag of zero (0).

[0038] Now refer to Figures 2A-2D , Figures 2A-2D Example annotations applied to sensor data for use in ground truth generation for training machine learning models to track objects are shown, according to some embodiments of the present disclosure. For example, Figure 2A Shows the Figure 1 The training process 100 generates images 200A of ground truth data. Figure 2B An example of annotations applied to sensor data to train a machine learning model to track an object is shown. Although images 200A and 200B are depicted as different images, this distinction is provided to facilitate description of particularly relevant features. In other words, in some aspects, the annotations (e.g., boundary shapes) depicted in image 200B can be included in image 200A.

[0039] Objects in an image (e.g., objects 202, 204, 206, 208, and 210) can be represented by boundary shapes (e.g., Figure 2B 222, 224, 226, 228, and 230) and track ID labels (e.g., track ID labels 232, 234, 236, 238, and 240). For example, object 202 can be bounded by boundary shape 222 and labeled with track ID 232. Similarly, object 204 can be annotated with boundary shape 224 and labeled with track ID 234, object 206 can be annotated with boundary shape 226 and labeled with track ID 236, object 208 can be annotated with boundary shape 228 and labeled with track ID 236, object 208 can be annotated with boundary shape 228 and labeled with track ID 238, and object 210 can be annotated with boundary shape 230 and labeled with track ID 240. Additionally, in some aspects, boundary shapes, objects, and / or pixels can be annotated with object classifications (e.g., vehicle, pedestrian, cyclist, etc.).

[0040] Reference again Figure 1 , the encoder 116 can be configured to encode ground truth information corresponding to the trajectory of an object through time and / or space using one or more annotations 110. For example, as described herein, a set of sensor data can be annotated with annotations 112. Once annotated, the set of data can be encoded by an encoder to generate ground truth data (e.g., trajectory ID map 118), which can include trajectory ID map 118. In addition, because the spatial resolution of the sensor data 102 applied to one or more machine learning models 104 may be different - e.g., greater than, less than, etc. - than the output spatial resolution corresponding to the prediction of the machine learning model 104, the ground truth information can be encoded into various formats. For example, an object identified during annotation can be tracked through a data sequence (e.g., a set of sensor data) to determine the object trajectory length (e.g., number of frames) of the annotated object. Based on the object trajectory, the data sequence can be filtered during encoding. For example, object trajectories shorter than a predetermined minimum trajectory length are removed during encoding. The predetermined minimum trajectory length can vary based on the object class (e.g., vehicle or pedestrian).

[0041] Additionally, filtering and / or sampling of the sensor data 102 may occur during encoding. For example, the last 30 frames of a data sequence (e.g., video) may be filtered to improve the accuracy of the ground truth data by eliminating object detection errors. For another example, to reduce the training burden, data sequences longer than a predetermined threshold may be sampled every n frames to reduce the data sequence length.

[0042] Furthermore, each pixel within a boundary shape can be encoded with a trajectory ID corresponding to the trajectory ID of the boundary shape. For example,Figure 2C and Figure 2D Two encoding examples of images that can be used to train a DNN are depicted. In some aspects, image 200C and image 200D can depict images captured by the same sensor at different times (e.g., tx, tx-1). In some aspects, image 200C and image 200D can depict images captured by different sensors (e.g., sensor 1 and sensor 2) at the same time.

[0043] Figure 2C A first encoded image 200C with detected objects defined by boundary shapes 242 and 244 is depicted. As described above, boundary shapes 242 and 244 can each have a track ID annotation. During encoding, each pixel within boundary shape 242 (e.g., vl, v2, v9, vlO, vl7, vl8) can be encoded with the track ID associated with boundary shape 242. For example, pixel 248 can be encoded with the same track ID as boundary shape 242. Similarly, each pixel within boundary shape 244 can be encoded with the track ID associated with boundary shape 244. For example, pixel 250 can be encoded with the same track ID as boundary shape 244. In some aspects, background pixels (e.g., pixels not covered by a boundary shape, such as pixel 246) can be assigned a common track ID (e.g., a track ID value of zero (0)).

[0044] Figure 2D A second encoded image 200D with detected objects defined by boundary shapes 262 and 264 is depicted. During encoding, each pixel within boundary shape 262 (e.g., v8, v9, vl6, vl7, v24, v25) can be encoded with the track ID associated with boundary shape 262. As described above, in some aspects, boundary shape 242 and boundary shape 262 can define the same object in images captured at different times. In some aspects, boundary shape 242 and boundary shape 262 can define the same object captured by different sensors at the same time. Accordingly, the track IDs encoded for pixels within boundary shape 242 and boundary shape 262 can be the same track ID.

[0045] Returning to Figure 1, once ground truth data (e.g., one or more track ID maps 118) is generated for each instance of sensor data 102 (e.g., for each image that sensor data 102 includes image data). For example, encoder 116 may encode multiple annotated images (e.g., image 200C and image 200D) to generate track ID map 118. The track ID map may thus include a series of discrete representations of sensor data. In some embodiments, the series may include discrete representations of sensor data over time and / or discrete representations of sensor data across multiple sensors. Additionally, the one or more track ID maps include ground truth object identifications (e.g., via track ID values) through the discrete representations of the sensor data. One or more machine learning models 104 may be trained using the ground truth data (e.g., track ID map 118). For example, one or more machine learning models 104 may generate feature descriptor maps 106, and the output may be compared to the ground truth data corresponding to each instance of sensor data 102. For example, the machine learning model 104 can generate an output feature descriptor map 106 (which can include a feature vector corresponding to each pixel at the output resolution), and the feature vectors from the feature descriptor map 106 can be compared using the track IDs corresponding to those same pixels from the track ID map 118. Thus, in cases where the feature vectors corresponding to two pixels are different, but the two pixels are known to correspond to the same object based on the track ID, one or more loss functions 126 can penalize the output so that the one or more machine learning models 104 learn to calculate more similar (e.g., identical) feature vectors for pixels corresponding to the same object. Similarly, for pixels corresponding to different objects, if the feature vectors are similar, but the ground truth data (e.g., track ID map 118) indicates that the pixels correspond to different objects, the machine learning model 104 may be penalized. Thus, feedback from the loss function 126 can be used to update the parameters (e.g., weights and biases) of the machine learning model 104 based on the ground truth data (e.g., track ID map 118) until the machine learning model 104 converges to an acceptable or desired accuracy. Using this process, one or more machine learning models 104 may be trained to accurately predict outputs from sensor data 102 (eg, one or more feature descriptor maps 106 ) using loss function 126 and ground truth data (eg, trajectory ID map 118 ).

[0046] The machine learning model 104 can use the sensor data 102 to compute an output (e.g., a feature descriptor map 106), which can ultimately be applied to a decoder or one or more other post-processing components (at least as described herein). Figure 5In more detail, the one or more machine learning models 104 are described in more detail below with respect to FIGS. 2-4 to generate object tracking information. Although the examples described herein are with respect to using a deep neural network (DNN), and in particular a convolutional neural network (CNN), as the machine learning model 104 (e.g., with respect to FIGS. 2-4), this is not intended to be limiting. For example, and without limitation, the machine learning model 104 can include any type of machine learning model, such as one or more machine learning models that use linear regression, logistic regression, decision trees, support vector machines (SVMs), naive Bayes, k-nearest neighbors (Knn), K-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoders, convolutional, recurrent, perceptron, long / short-term memory (LSTM), hopfield, Boltzmann, deep belief, deconvolutional, generative adversarial, liquid machines, etc.), computer vision algorithms, and / or other types of machine learning models. Figure 1 and Figure 5 ), but this is not intended to be limiting. For example, and without limitation, the machine learning model 104 can include any type of machine learning model, such as one or more machine learning models that use linear regression, logistic regression, decision trees, support vector machines (SVMs), naive Bayes, k-nearest neighbors (Knn), K-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoders, convolutional, recurrent, perceptron, long / short-term memory (LSTM), hopfield, Boltzmann, deep belief, deconvolutional, generative adversarial, liquid machines, etc.), computer vision algorithms, and / or other types of machine learning models.

[0047] As an example, in the case where the one or more machine learning models 104 include a CNN, for example, the one or more machine learning models 104 can include any number of layers. One or more of the layers can include an input layer. The input layer can hold values associated with the sensor data 102 (e.g., before or after post-processing). For example, when the sensor data 102 is an image, the input layer can hold values representing the raw pixel values of one or more images as a volume (e.g., width, height, and color channels (e.g., RGB), such as 32x32x3).

[0048] One or more of the layers can include a convolutional layer. The convolutional layer can compute the output of neurons connected to local regions in the input layer, each neuron computing their weighted sum of the dot product between their weights and the small region they are connected to in the input volume. The result of the convolutional layer can be another volume, where one dimension is based on the number of filters applied (e.g., width, height, and number of filters, such as 32x32x12, if 12 is the number of filters).

[0049] One or more of the layers can include a rectified linear unit (ReLU) layer. The ReLU layer can apply an element-wise activation function, such as max(0, x), for example, zero thresholding. The resulting volume of the ReLU layer can be the same volume as the input to the ReLU layer.

[0050] One or more of the layers can include a pooling layer. The pooling layer can perform a down-sampling operation along the spatial dimensions (e.g., height and width), which can result in a smaller volume than the input to the pooling layer (e.g., 16x16x12 from a 32x32x12 input volume).

[0051] One or more layers can include one or more fully connected layers. Each neuron in the one or more fully connected layers can be connected to every neuron in the previous convolution. The fully connected layers can compute class scores, and the resulting volume can be 1 x 1 x number of classes. In some examples, a CNN can include one or more fully connected layers such that the output of one or more layers of the CNN can be provided as input to one or more fully connected layers of the CNN. In some examples, one or more convolutional streams can be implemented by one or more machine learning models 104, and some or all of the convolutional streams can include respective one or more fully connected layers.

[0052] In some non-limiting embodiments, one or more machine learning models 104 can include a series of convolution and max pooling layers to facilitate image feature extraction, followed by multi-scale dilated convolution and up-sampling layers to facilitate global contextual feature extraction.

[0053] While input layers, convolutional layers, pooling layers, ReLU layers, and fully connected layers are discussed herein with respect to machine learning models 104, this is not intended to be limiting. For example, additional or alternative layers can be used in one or more machine learning models 104, such as normalization layers, SoftMax layers, and / or other layer types.

[0054] In embodiments in which one or more machine learning models 104 include a CNN, different orders and numbers of CNN layers can be used depending on the embodiment. In other words, the order and number of layers of one or more machine learning models 104 are not limited to any one architecture.

[0055] Further, some layers can include parameters (e.g., weights and / or biases), such as convolutional layers and fully connected layers, while others can not, such as ReLU layers and pooling layers. In some examples, the parameters can be learned by one or more machine learning models 104 during training. Further, some layers can include additional hyperparameters (e.g., learning rate, stride, epoch, etc.), such as convolutional layers, fully connected layers, and pooling layers, while others can not, such as ReLU layers. The parameters and hyperparameters are not limited and can differ depending on the embodiment.

[0056] The process 100 includes triplets mining 120. The triplets mining 120 can include generating one or more vector triplets 122 from the training data. The one or more vector triplets 122 can be mined from feature descriptor maps (e.g., one or more feature descriptor maps 106 and / or one or more track ID maps 118) across space and / or time. For example, in a single-sensor tracking scenario, at least two descriptor maps (e.g., where the feature descriptor maps are generated from sensor data captured by a single sensor, the descriptor maps 200C can represent t 1-kAnd the descriptor graph 200D can represent t1) that can be used to create a vector triple. The vector triple can include a positive sample, a negative sample, and an anchor. The positive sample can be one or more vectors from the positive sample of the descriptor graph 200C (e.g., vectors within the bounding shape with the same trajectory ID), while the negative sample can be one or more vectors from the negative sample of the descriptor graph 200C (e.g., vectors within another bounding shape). The anchor can be one or more vectors from the positive sample of the descriptor graph 200D.

[0057] For another example, vector triples can be generated to train a DNN for cross-sensor tracking - e.g., at the same time step. In such an example, (e.g., in the case where the feature descriptor graphs 200C and 200D are generated by different sensors) the positive sample can be one or more vectors from the positive object of the descriptor graph 200C of the first sensor (e.g., vectors within the bounding shape 242), while the negative sample can be one or more vectors from the negative sample of the descriptor graph 200C of the first sensor (e.g., vectors within the bounding shape 244). The anchor can be one or more vectors from the bounding shape 262 of the descriptor graph 200D of the second sensor.

[0058] In other words, in some aspects, the vector triple can include pixel-level vector based feature descriptors identified by the machine learning model for at least two time points and / or positive objects from two sensors (e.g., the positive sample and the anchor). The vector triple can also include pixel-level vector based feature descriptors identified by the machine learning model for at least one time point and / or negative objects from one sensor (e.g., the negative sample).

[0059] Vector triples can be mined or generated in a variety of ways. For example and returning Figure 1 The sensor data 102 can be provided as input to one or more machine learning models 104. The one or more machine learning models 104 can output one or more feature descriptor graphs 106. In the case where the sensor data is an image or a series of images, the one or more feature descriptor graphs 106 can include one-dimensional (ID) vectors generated for each pixel in the image (at an output resolution of the one or more machine learning models 104). Moreover, since the spatial resolution of the sensor data 102 applied to the machine learning models 104 can differ - e.g., be greater than, less than, etc. - the output spatial resolution corresponding to the predictions of the machine learning models 104, the feature descriptor graphs can be encoded in a variety of formats. In some aspects, each vector associated with a pixel within a bounding shape (e.g., v n) are designated as positive vectors. In some aspects, each vector associated with a pixel inside the boundary shape in the corresponding trajectory ID map associated with the second object is designated as a negative vector. Further, in some aspects, each vector associated with a pixel inside the boundary shape in the second corresponding trajectory ID map associated with the first object is designated as an anchor vector.

[0060] Vector triplets can be created from the feature descriptor maps across space and / or time. For example, a positive vector (p), a negative vector (n), and an anchor vector (a) can be combined to form a set of vector triplets. For a given triplet, a triplet loss function can be used to minimize the distance between the anchor feature descriptor and the positive feature descriptor (e.g., distance “d” between anchor “a” and positive “p”). In other words, the triplet loss minimizes d(a, p). Similarly, the triplet loss function can minimize the distance between the anchor and the negative (e.g., minimize d(a, n)). In some embodiments, a margin value can be added to the maximum margin loss. For example, in a pseudo function expression, the loss function 124 can include a loss function of equation (1) as follows:

[0061]

[0062] Accordingly, feedback from the one or more loss functions 126 can be used to update the parameters (e.g., weights and biases) of the one or more machine learning models 104 according to the ground truth data (e.g., the one or more trajectory ID maps 118) until the one or more machine learning models 104 converge to an acceptable or desired accuracy. Using the process 100, the one or more machine learning models 104 can be trained to accurately predict the output (e.g., the one or more feature descriptor maps 106) (and / or associated classifications) from the sensor data 102 using the one or more loss functions 124, the one or more vector triplets 122, and the ground truth data (e.g., the one or more trajectory ID maps 118).

[0063] As described above, triplets are generally categorized into three types: easy triplets, hard triplets, and semi-hard triplets. The triplet loss for easy triplets is zero (0), hard triplets are triplets where the negative sample is closer to the anchor than the positive sample, and semi-hard triplets are triplets where the negative sample is not closer to the anchor than the positive sample, but still has a positive loss. In some embodiments, semi-hard triplets and hard triplets can be used during training as feedback from the one or more loss functions 126 to update the parameters of the one or more machine learning models 104. In some embodiments, in some aspects, easy, semi-hard, and hard triplets can be used during training.

[0064] Further, the vector triplets can be mined using round or batch mining. For example, for round mining, one or more vector triplets 122 can be generated for each feature descriptor map 106 based on one or more trajectory ID maps 120 for a round. Feedback from the loss function for each triplet from the round can be used to update the parameters. For another example, for batch mining one or more vector triplets 122, a subset of a round can be batched and, for each triplet of the batch, feedback from the one or more loss functions 124 can be used to update the parameters of the one or more machine learning models 104. Another subset of the round can be batched and feedback generated using the updated parameters of the one or more machine learning models 104. The one or more vector triplets 122 from the second subset and, for each triplet of the batch, feedback from the one or more loss functions 124 can be used to update the parameters of the one or more machine learning models 104. This batch mining approach can be used in batch-all or batch-next difficult contexts.

[0065] For example, and briefly turning to Figure 3A and with continued reference to Figure 1 According to some aspects described herein, an example feature descriptor map 300A is depicted. The feature descriptor map 300A includes positive vectors (e.g., v0, vl, v8, v9, vl6, and vl7) and negative vectors (e.g., vl3, vl4, vl5, v21, v22, v23, v29, v30, v31). The feature descriptor map 300A also includes a first boundary shape 302 and a second boundary shape 304. Briefly turning to Figure 3A and with continued reference to Figure 1 According to some aspects described herein, an example feature descriptor map 300B is depicted. The feature descriptor map 300B includes positive vectors (e.g., v9, vlO, vl7, vl8, v25, and v26) and negative vectors (e.g., v5, v6, v7, vl3, vl4, vl5, v21, v22, and v23). In particular, the feature descriptor map 300B includes positive vectors generated at different times from the same object and / or detected by different sensors as the positive vectors from the feature descriptor map 300A. Thus, for vector triplet mining, the feature descriptor map 300B can be said to include anchor vectors (e.g., v9, vlO, vl7, vl8, v25, and v26).

[0066] In batch all triplets mining, all valid triplets are selected for a given batch. For example, valid triplets with anchor v9 from feature descriptor graph 300B include any positive vectors from feature descriptor graph 300A (e.g., v0, vl, v8, v9, vl6, and vl7) and any negative vectors from either feature descriptor graph 300A or 300B. In some embodiments, a loss is determined for each valid triplet mined. In some embodiments, hard triplets and semi-hard triplets can be averaged, and simple triplets are ignored. In batch hard triplet mining, the hardest positive and hardest negative in the batch for each anchor are selected, and the triplet loss function is computed only for the hardest triplets.

[0067] Furthermore, those skilled in the art will appreciate that Figure 2C , Figure 2D , Figure 3A , Figure 3B , Figure 6A and 6B The boundary shapes and pixels depicted in FIGS. 1-8 are illustrative. As shown, the pixels are not enclosed by the boundary shapes. In other words, it is recognized that a portion of the pixels extend beyond the Figure 2C , Figure 2D , Figure 3A , Figure 3B , Figure 6A and 6B The boundary shapes and pixel boundaries depicted in FIGS. 1-8 are illustrative. As shown, the pixels are not enclosed by the boundary shapes. In other words, it is recognized that a portion of the pixels extend beyond the

[0068] Referring now to FIGS. 1-8, Figure 4A and 4B Each block of the methods 400 and 420 described herein comprises a computational procedure that can be executed using any combination of hardware, firmware, and / or software. For example, various functions can be carried out by a processor executing instructions stored in memory. The methods 400 and 420 can also be embodied as computer-usable instructions stored on computer storage media. The methods 400 and 420 can be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service) or a plug-in to another product, to name a few. Figure 1 The methods 400 and 420 are described by way of example with respect to the process 100 of FIG. 1. However, the methods 400 and 420 can additionally or alternatively be performed by any one system or any combination of systems, including but not limited to those described herein.

[0069] Figure 4Ais an example flowchart illustrating a method 400 for training a neural network to track objects, in accordance with some embodiments of the present disclosure. At block B402, the method 400 includes receiving sensor data. For example, the sensor data 102 can be generated and / or received, where the sensor data 102 includes image data representing images depicting locations of objects at different times and / or from different sensors.

[0070] At block B404, the method 400 includes receiving first data representing annotations corresponding to bounding shapes associated with objects represented by the sensor data, and second data corresponding to track ID labels for the bounding shapes. For example, data representing the object detections 110 and the annotations 112 can be generated and / or received.

[0071] At block B406, the method 400 includes encoding a track ID label associated with each bounding shape to each pixel defined by the bounding shape. For example, the encoder 116 can encode pixels having the track ID labels to the track ID maps 120 to use as ground truth data (e.g., the track ID maps 118) to train the one or more machine learning models 104.

[0072] At block B408, the method 400 includes mining multi-vector triplets from the one or more feature descriptor maps. For example, the vector triplets 122 can be mined based on the one or more track ID maps 120 and the one or more feature descriptor maps 106. The vector triplets can be one of three types: simple triplets, hard triplets, and semi-hard triplets. The loss for simple triplets is zero (0), hard triplets are triplets.

[0073] At block 410, the method 400 includes training a neural network using the one or more feature descriptor maps as ground truth data. For example, the one or more track ID maps 120 can be used as ground truth data, along with the one or more vector triplets 122, to train a machine learning model to track objects.

[0074] Figure 4B is another example flowchart illustrating a method 420 for training a neural network to track objects, in accordance with some embodiments of the present disclosure. At block B422, the method 420 includes labeling a first bounding shape of a first frame and a second bounding shape of a second frame as corresponding to an object using training data. For example, the object detection 108 can label the sensor data 102 with the bounding shapes 110.

[0075] At block B424, the method 420 includes determining, using the deep neural network, one or more first feature descriptors corresponding to a first point within a first boundary shape of the first frame and one or more second feature descriptors corresponding to a second point within a second boundary shape of the second frame. For example, the triplet mining 120 can be used to generate anchor vectors of the vector triplets 122 based on the feature descriptor map 106 and the corresponding trajectory ID map 118.

[0076] At block B426, the method 420 includes determining a first feature descriptor of the one or more first feature descriptors corresponding to an anchor point of the first point. For example, the triplet mining 120 can be used to generate anchors of the vector triplets 122 based on the feature descriptor map 106 and the corresponding trajectory ID map 118.

[0077] At block B428, the method 420 includes comparing, using a loss function, the first feature descriptor and a second feature descriptor corresponding to a second point of the second point. For example, the loss function 124 can be used to compare the feature descriptors of the vector triplets 122.

[0078] At block B430, the method 420 includes updating one or more parameters of the deep neural network based at least in part on the comparison. For example, the output of the loss function 124 can be used to update one or more parameters of the machine learning model 104.

[0079] Reference is now made to Figure 5 , Figure 5 is a dataflow diagram illustrating an example process 500 for generating object tracking data in real-time or near real-time using output from sensors of a vehicle, in accordance with some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are set forth only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, groupings of functions, etc.) can be used in addition to or instead of those shown, and some elements can be wholly omitted depending on the context. Further, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in conjunction with other components, and in any suitable combination and location. The various functions described herein as being performed by entities can be performed in hardware, firmware, and / or software. For instance, various functions can be performed through execution of instructions stored in memory by a processor. In some embodiments, the systems, methods, and processes described herein can be performed using components, features, and / or functionality similar to those of the example autonomous vehicle 800 of FIGS. 8A-8D, Figure 9 the example computing device 900 of FIGS. 9A-9D, and / or Figure 10 the example data center 1000 of FIGS. 10A-10D. While the tracked objects described with respect to Figure 5 are vehicles, this is not intended to be limiting and is used for example purposes only.

[0080] Sensor data 102 can be similar to that described herein at least with respect to Figure 1 During deployment, sensor data 102 can be applied to one or more machine learning models 104 that have been trained according to process 100, e.g., to compute one or more feature descriptor maps 106 using the training sensor data 102. Accordingly, one or more machine learning models 104 can output - or regress to - one or more feature descriptor maps 106 as described in more detail with respect to one or more feature descriptor maps 108 at least with respect to Figure 1

[0081] One or more feature descriptor maps 106 can be applied to decoder 502 to generate one or more decoded outputs 504. In some examples, decoder 502 can use one or more post-processing algorithms (e.g., temporal smoothing, curve fitting, filtering, etc.) to compute one or more decoded outputs 504. For example, decoder 502 can use one or more feature descriptor maps 106 to track objects through time and / or across multiple sensors, which can be used in turn to generate control commands for ego machines, e.g., through control component 508. Decoded outputs can include object IDs, pixel-level feature vectors, boundary shape-level feature vectors, and / or other information corresponding to detected objects (e.g., boundary shapes corresponding to objects, which can be used to generate visualizations of objects across time and / or across sensors for debugging, etc.).

[0082] In some embodiments, one or more outputs can be generated at a first spatial resolution (e.g., a down-sampled spatial resolution) that is different than a second spatial resolution corresponding to sensor data 102. Accordingly, decoder 502 can decode data from the first spatial resolution of the output and project or attribute the data to the second spatial resolution of sensor data 102 (e.g., using one or more feature descriptor maps 106 as described herein). As a result, processing speed of one or more machine learning models 104 can be increased while reserving computational resources for other tasks of the system. Moreover, by increasing processing speed, run-time of the system can be reduced, enabling real-time or near real-time deployment of process 500 within a system (e.g., vehicle 800).

[0083] ​Additionally or alternatively, the output can be used to facilitate object tracking with relatively minimal downstream impact, which can reserve computational resources for other tasks of the system. For example, in some embodiments, pixel-level feature descriptors can be converted to boundary shape-level feature descriptors during post-processing 504. Post-processing 504 can include averaging pixel-level feature descriptors within a particular boundary shape of the feature descriptor map 106. The averaged feature descriptors can then be assigned as boundary shape-level feature descriptors. The boundary shape-level feature descriptors can be used to track corresponding objects across frames (from the same sensor or across different sensors).

[0084] For example, referring to Figure 6A For the feature descriptor map 600, the pixel-level feature descriptor vectors associated with the first boundary shape (e.g., boundary shape 602, as computed using the object detection module 510) (e.g., v0, vl, v8, v9, vl6, and vl7) can be averaged and the resulting feature descriptor vector can be assigned as an object feature descriptor vector. Similarly, the pixel-level feature descriptor vectors associated with the second boundary shape (e.g., boundary shape 604) (e.g., vl3, vl4, vl5, vl, vl2, vl3, v29, v30, and v31) can be averaged and the resulting feature descriptor vector can be assigned as an object feature descriptor vector. In embodiments, the boundary shapes corresponding to the objects can be computed using the object detection module 510, which can include a computer vision algorithm, a machine learning model, a neural network (e.g., a convolutional neural network), and / or another type of algorithm that computes boundary shape locations. In some embodiments, the object detection module 510 can be included within the one or more machine learning models 104 - e.g., the one or more machine learning models 104 can compute the predicted feature descriptor map 106 in addition to the locations of the boundary shapes.

[0085] Additionally or alternatively, the post-processing 504 can generate tracking output 506 to facilitate tracking objects across time and / or sensors. For example, and referring to Figure 6BIn some applications of single-sensor tracking over a time range, the machine learning model 104 can process consecutive frames separately and generate feature descriptor maps (e.g., feature descriptor maps 610, 620, 630, and 640) that include the boundary shape level feature descriptors of detected objects as described above. An object association score can be generated between each boundary shape level feature descriptor of the time series. For example, an object association score can be generated between each combination of boundary shape level feature descriptors 612, 614, 622, 624, 632, 634, 642, and 644 using the cosine distance between vectors. The difference of the feature descriptors can be computed and the best match can be identified as the same object across the time series. For example, the best match can be found using a K-Nearest Neighbor (KNN) algorithm for tracking purposes.

[0086] Additionally or alternatively, in some embodiments, the boundary shape level feature descriptors of each object (e.g., 612, 622, and 632) can be averaged and can be assigned to a trajectory of the object over the time series. The boundary shape level feature descriptors (e.g., 642) of the most recent feature descriptor map (e.g., feature descriptor map 640) can be compared to each averaged feature descriptor of the trajectory. The best match between the most recent feature descriptor and the averaged feature descriptors is assigned to the same object trajectory.

[0087] Similarly, referring briefly to Figure 6C In some applications of cross-sensor tracking, one or more machine learning models 104 can process frames generated by each sensor separately and generate feature descriptor maps (e.g., feature descriptor maps 650, 660, 670, and 680) that include the boundary shape level feature descriptors of detected objects as described above. An object association score can be generated between each boundary shape level feature descriptor of the multi-sensor series. For example, an object association score can be generated between each combination of boundary shape level feature descriptors 652, 654, 662, 664, 672, 674, 682, and 684 using the cosine distance between vectors. The difference of the feature descriptors can be computed and the best match can be identified as the same object across the time series. For example, the best match can be found using a K-Nearest Neighbor (KNN) algorithm for tracking purposes.

[0088] Additionally or alternatively, in some embodiments, the boundary shape level feature descriptors of each object (e.g., 652, 662, and 672) across sensors can be averaged and can be assigned to a trajectory of the object. The boundary shape level feature descriptors (e.g., 682) of the feature descriptor map (e.g., feature descriptor map 680) of the other sensor can be compared to each averaged feature descriptor of the trajectory. The best match between the most recent feature descriptor and the averaged feature descriptors is assigned to the same object trajectory.

[0089] Accordingly, the sensor data 102 can be used as input to one or more machine learning models. The one or more machine learning models can assign an identity to objects present in the sensor data and can use the identity to track the objects across multiple sensors and / or over time. The tracking output 506 can be communicated to a control component 508 of the autonomous machine (e.g., ego machine). For example, the control component 508 can use this data to brake, accelerate, change path through steering input, or any other control operation that the autonomous machine is capable of.

[0090] Referring now to Figure 7 , each block of the method 700 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For instance, various functions can be carried out by a processor executing instructions stored in memory. The method 700 can also be embodied as computer-usable instructions stored on computer storage media. The method 700 can be provided by a stand-alone application, a service or hosted service (stand-alone or in combination with another hosted service), or a plug-in to another product, to name a few. Figure 5 The method 700 is described with respect to the process 500 of

[0091] Figure 7 is a flowchart illustrating a method 700 for detecting and tracking objects according to some embodiments of the present disclosure. Although not depicted, some embodiments of the method 700 receive sensor data and apply the sensor data to a neural network. For example, the sensor data 102 representing object images in a field of view of at least one sensor of the vehicle 800 can be applied to one or more machine learning models 104.

[0092] At block B702, the method 700 includes computing, using a neural network and based at least in part on sensor data of an ego machine, a first feature map representing one or more feature descriptors. For example, the machine learning model 104 can compute one or more outputs including one or more feature descriptor maps 106.

[0093] At block B704, the method 700 includes associating, based at least in part on the one or more first feature descriptors, the first feature descriptors with a first object. For example, the decoder 502 can decode the one or more feature descriptor maps 106 to determine a boundary shape level feature descriptor vector for each object detected in the sensor data. The decoder 502 can determine the feature descriptor vector for the boundary shape based on a computed average of the pixel level feature descriptor vectors.

[0094] At block B706, the method 700 includes computing, using the deep neural network and based at least in part on second sensor data generated using one or more sensors of the ego machine, a second feature map representing one or more second feature descriptors. For example, the machine learning model 104 can compute one or more outputs including one or more second feature descriptor maps 106.

[0095] At block B708, the method 700 includes associating the second feature descriptor with a second object based at least in part on the one or more second feature descriptors. For example, the decoder 502 can decode the one or more feature descriptor maps 106 to determine a boundary shape level feature descriptor vector for each object detected in the sensor data. The decoder 502 can determine the feature descriptor vector for the boundary shape based on a computed average of the pixel level feature descriptor vectors.

[0096] At B710, the method 700 includes determining that the first object and the second object are the same object. For example, the decoder 502 can determine a best match between the boundary shape level feature descriptors based on the computed object association scores. The best match between the boundary shapes can be identified as the same object across sensors and / or time in the tracking output 506.

[0097] At block B712, the method 700 includes performing, by the vehicle, one or more operations based on determining that the first object and the second object are the same object. For example, the one or more control components 508 can cause the vehicle 800 to perform one or more operations based on a suggested path determined based on the tracking output 506.

[0098] Example autonomous vehicle

[0099] Figure 8Ais a diagram of an example autonomous vehicle 800 according to some embodiments of the present disclosure. Autonomous vehicle 800 (alternatively referred to herein as “vehicle 800”) can include, but is not limited to, a passenger vehicle such as a car, truck, bus, first responder vehicle, shuttle, electric or motorized bicycle, motorcycle, fire vehicle, police vehicle, ambulance, boat, construction vehicle, underwater vessel, drone, vehicle connected to a trailer, and / or another type of vehicle (e.g., a vehicle that is unmanned and / or that accommodates one or more passengers). Autonomous vehicles are generally described in terms of levels of automation as defined by a department of the United States Department of Transportation, the National Highway Traffic Safety Administration (NHTSA), and the Society of Automotive Engineers (SAE) “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles” (Standard No. J3016-201806 published June 15, 2018, Standard No. J3016-201609 published September 30, 2016, and prior and future versions of this standard). Vehicle 800 can be capable of implementing one or more of Levels 3-5 of autonomous driving. Vehicle 800 can be capable of implementing one or more of Levels 1-5 of autonomous driving. For example, depending on the embodiment, vehicle 800 can be capable of implementing driver assistance (Level 1), partial automation (Level 2), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5). As used herein, the term “autonomous” can include any and / or all types of autonomy of vehicle 800 or other machines, such as fully autonomous, highly autonomous, conditionally autonomous, partially autonomous, providing assistance autonomous, semi-autonomous, primarily autonomous, or other designations.

[0100] Vehicle 800 can include components such as a chassis, a body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components of a vehicle. Vehicle 800 can include a propulsion system 850, such as an internal combustion engine, a hybrid power plant, an all-electric motor, and / or another type of propulsion system. Propulsion system 850 can be connected to a drivetrain of vehicle 800 that can include a transmission in order to effectuate propulsion of vehicle 800. Propulsion system 850 can be controlled in response to receiving a signal from a throttle / accelerator 852.

[0101] A steering system 854, which can include a steering wheel, can be used to steer the vehicle 800 (e.g., along a desired path or route) while the propulsion system 850 is operating (e.g., while the vehicle is in motion). The steering system 854 can receive signals from a steering actuator 856. For full automation (level 5) functionality, the steering wheel can be optional.

[0102] A braking sensor system 846 can be used to operate the vehicle brakes in response to receiving signals from the braking actuator 848 and / or braking sensors.

[0103] One or more controllers 836, which can include one or more system on a chip (SoC) 804 Figure 8C ) and / or one or more GPUs, can provide signals (e.g., representing commands) to one or more components and / or systems of the vehicle 800. For example, the one or more controllers can send signals to operate the vehicle brakes via the one or more braking actuators 848, to operate the steering system 854 via the one or more steering actuators 856, to operate the propulsion system 850 via the one or more throttle / accelerator 852. The one or more controllers 836 can include one or more on-board (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operational commands (e.g., signals representing commands) to enable autonomous driving and / or to assist a human driver in driving the vehicle 800. The one or more controllers 836 can include a first controller 836 for autonomous driving functionality, a second controller 836 for functional safety functionality, a third controller 836 for artificial intelligence functionality (e.g., computer vision), a fourth controller 836 for infotainment functionality, a fifth controller 836 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 836 can handle two or more of the above functionalities, two or more controllers 836 can handle a single functionality, and / or any combination thereof.

[0104] One or more controllers 836 can provide signals for controlling one or more components and / or systems of the vehicle 800 in response to sensor data (e.g., sensor inputs) received from one or more sensors. The sensor data can be received from, for example and without limitation, global navigation satellite system sensors 858 (e.g., global positioning system sensors), RADAR sensors 860, ultrasonic sensors 862, LIDAR sensors 864, inertial measurement unit (IMU) sensors 866 (e.g., accelerometers, gyroscopes, magnetic compasses, magnetometers, etc.), microphones 896, stereo cameras 868, wide-angle cameras 870 (e.g., fisheye cameras), infrared cameras 872, surround cameras 874 (e.g., 360 degree cameras), long and / or medium range cameras 898, speed sensors 844 (e.g., to measure the speed of the vehicle 800), vibration sensors 842, steering sensors 840, brake sensors (e.g., as part of a brake sensor system 846), and / or other sensor types.

[0105] One or more of the controllers 836 can receive inputs (e.g., represented by input data) from the instrument cluster 832 of the vehicle 800 and provide outputs (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display 834, audible annunciators, speakers, and / or via other components of the vehicle 800. These outputs can include information such as vehicle speed, velocity, time, map data (e.g., an HD map 822 of the Figure 8C

[0106] The vehicle 800 also includes a network interface 824 that can communicate over one or more networks using one or more wireless antennas 826 and / or modems. For example, the network interface 824 can be capable of communicating over LTE, WCDMA, UMTS, GSM, CDMA2000, etc. The one or more wireless antennas 826 can also enable communication between objects (e.g., vehicles, mobile devices, etc.) in the implementation environment using one or more local area networks such as Bluetooth, Bluetooth LE, Z-Wave, ZigBee, etc. and / or one or more low power wide area networks (LPWANs) such as LoRaWAN, SigFox, etc.

[0107] Figure 8B ​To provide an example autonomous vehicle 800 according to some embodiments of the present disclosure Figure 8A An example of camera locations and fields of view of the example autonomous vehicle 800. The cameras and respective fields of view are one example embodiment and are not intended to be limiting. For example, additional and / or alternative cameras can be included and / or these cameras can be located at different locations on the vehicle 800.

[0108] Camera types for the cameras can include, but are not limited to, digital cameras that can be suitable for use with components and / or systems of the vehicle 800. The cameras can operate at Automotive Safety Integrity Level (ASIL) B and / or at another ASIL. The camera types can have any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc., depending on the embodiment. The cameras can be capable of using a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, a color filter array can include a red- white-white-white (RCCC) color filter array, a red-white-white-blue (RCCB) color filter array, a red-blue-green-white (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, clear pixel cameras, such as cameras with a

[0109] In some examples, one or more of the cameras can be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-functional mono camera can be installed to provide functions including lane departure warning, traffic sign assist, and intelligent headlamp control. One or more (e.g., all) of the cameras can simultaneously record and provide image data (e.g., video).

[0110] One or more of the cameras can be installed in mounting assemblies, such as custom designed (3-D printed) assemblies, in order to cut off stray light and reflections from within the car (e.g., reflections from the dashboard reflected in the windshield mirror) that can interfere with the image data capture capabilities of the cameras. With regard to wing mirror mounting assemblies, the wing mirror assemblies can be custom 3-D printed such that the camera mounting plates match the shape of the wing mirrors. In some examples, one or more cameras can be integrated into the wing mirrors. For side view cameras, one or more cameras can also be integrated into the four pillars at each corner of the cab.

[0111] Cameras with fields of view that include portions of the environment in front of the vehicle 800 (e.g., front-facing cameras) can be used for surround view to help identify the forward path and obstacles, and to assist in providing information critical to generating an occupancy grid and / or determining a preferred vehicle path with the help of one or more controllers 836 and / or control SoCs. Front-facing cameras can be used to perform many of the same ADAS functions as LIDAR, including emergency braking, pedestrian detection, and collision avoidance. Front-facing cameras can also be used for ADAS functions and systems, including lane departure warning (LDW), adaptive cruise control (ACC), and / or other functions such as traffic sign recognition.

[0112] A wide variety of cameras can be used in a front-facing configuration, including, for example, monocular camera platforms including CMOS (complementary metal-oxide semiconductor) color imagers. Another example can be a wide-angle camera 870, which can be used to perceive objects (e.g., pedestrians, intersection traffic, or bicycles) entering the field of view from the periphery. Although Figure 8B Although only one wide-angle camera is illustrated in FIG. 8, any number of wide-angle cameras 870 can be present on the vehicle 800. In addition, long-range cameras 898 (e.g., long-view stereo camera pairs) can be used for depth-based object detection, especially for objects for which a neural network has not been trained. Long-range cameras 898 can also be used for object detection and classification and basic object tracking. For example, one or more long-range cameras 898 can generate sensor data 102.

[0113] One or more stereo cameras 868 can also be included in a front-facing configuration. Stereo cameras 868 can include an integrated control unit that includes a scalable processing unit that can provide a multi-core microprocessor with integrated CAN or Ethernet interfaces and programmable logic (FPGA) on a single chip. Such a unit can be used to generate a 3-D map of the vehicle’s environment, including distance estimates for all points in the image. Alternative stereo cameras 868 can include a compact stereo vision sensor that can include two camera lenses (one on the left and one on the right) and an image processing chip that can measure the distance from the vehicle to a target object and use the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 868 can be used in addition to or instead of those described herein.

[0114] Cameras with fields of view that include portions of the environment on the sides of the vehicle 800 (e.g., side-view cameras) can be used for surround view to provide information used to create and update an occupancy grid and to generate side impact collision warnings. For example, surround cameras 874 (e.g., as illustrated in FIG. 8) can be used to provide a 360-degree view of the vehicle’s environment. The surround cameras 874 can be used to provide a 360-degree view of the vehicle’s environment, which can be used to create and update an occupancy grid and to generate side impact collision warnings. For example, the surround cameras 874 can be used to provide a 360-degree view of the vehicle’s environment, which can be used to create and update an occupancy grid and to generate side impact collision warnings. Figure 8BFour surround cameras 874) can be placed on the vehicle 800. The surround cameras 874 can include wide-view cameras 870, fisheye cameras, 360-degree cameras, and / or the like. In one example, four fisheye cameras can be placed on the front, back, and sides of the vehicle. In an alternative arrangement, the vehicle can use three surround cameras 874 (e.g., left, right, and back) and can utilize one or more other cameras (e.g., a forward-facing camera) as a fourth surround camera.

[0115] Cameras with fields of view that include portions of the environment behind the vehicle 800 (e.g., rearview cameras) can be used to assist with parking, surround view, rear collision warnings, and creating and updating the occupancy grid. A wide variety of cameras can be used, including but not limited to cameras that are also suitable as front-facing cameras (e.g., long and / or mid-range cameras 898, stereo cameras 868, infrared cameras 872, etc.) as described herein.

[0116] As will be appreciated, one or more long-range cameras 898, one or more surround cameras 874, one or more stereo cameras 868, one or more infrared cameras 872, or any other sensors can generate sensor data 102, which can be provided as input to one or more machine learning models (e.g., regarding Figure 1 and Figure 5 The one or more machine learning models 104 described.

[0117] Figure 8C FIG. 1 is a block diagram of an example system architecture for an example autonomous vehicle 800 in accordance with some embodiments of the present disclosure. Figure 8A FIG. 1 is a block diagram of an example system architecture for an example autonomous vehicle 800 in accordance with some embodiments of the present disclosure.

[0118] Figure 8CEach of the components, features, and systems of the vehicle 800 are illustrated as being connected via a bus 802. The bus 802 can include a controller area network (CAN) data interface (alternatively referred to herein as a "CAN bus"). The CAN can be a network within the vehicle 800 that is used to assist in controlling various features and functions of the vehicle 800, such as the actuation of brakes, acceleration, braking, steering, windshield wipers, and so forth. The CAN bus can be configured to have tens or even hundreds of nodes, each with its own unique identifier (e.g., CAN ID). The CAN bus can be read to find steering wheel angle, ground speed, revolutions per minute (RPM) of the engine, button positions, and / or other vehicle status indicators. The CAN bus can be ASIL B compliant.

[0119] Although the bus 802 is described herein as a CAN bus, this is not intended to be limiting. For example, FlexRay and / or Ethernet can be used in addition to or alternatively from the CAN bus. Further, although the bus 802 is represented with a single line, this is not intended to be limiting. For example, there can be any number of buses 802, which can include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses that use different protocols. In some examples, two or more buses 802 can be used to perform different functions, and / or can be used for redundancy. For example, a first bus 802 can be used for collision avoidance functions, and a second bus 802 can be used for drive control. In any example, each bus 802 can communicate with any component of the vehicle 800, and two or more buses 802 can communicate with the same components. In some examples, each SoC 804, each controller 836, and / or each computer within the vehicle can have access to the same input data (e.g., input from sensors of the vehicle 800), and can be connected to a common bus, such as a CAN bus.

[0120] The vehicle 800 can include one or more controllers 836, such as those described herein with respect to Figure 8A The controllers 836 can be used for a wide variety of functions. The controllers 836 can be coupled to any of the other different components and systems of the vehicle 800, and can be used for control of the vehicle 800, artificial intelligence of the vehicle 800, infotainment for the vehicle 800, and / or the like.

[0121] The vehicle 800 can include one or more system on a chip (SoC) 804. The SoC 804 can include a CPU 806, a GPU 808, a processor 810, a cache 812, an accelerator 814, a data store 816, and / or other components and features not illustrated. The SoC 804 can be used to control the vehicle 800 in a wide variety of platforms and systems. For example, one or more SoCs 804 can be used in a system (e.g., a system of the vehicle 800) in conjunction with an HD map 822 that can be obtained from one or more servers (e.g., the one or more servers 878) via a network interface 824 for map refreshes and / or updates. Figure 8D

[0122] The CPU 806 can include a CPU cluster or CPU complex (alternatively referred to herein as a “CCPLEX”). The CPU 806 can include multiple cores and / or L2 caches. For example, in some embodiments, the CPU 806 can include eight cores in a coherent multi-processor configuration. In some embodiments, the CPU 806 can include four dual-core clusters with each cluster having a dedicated L2 cache (e.g., a 2 MB L2 cache). The CPU 806 (e.g., the CCPLEX) can be configured to support simultaneous cluster operation such that any combination of clusters of the CPU 806 can be active at any given time.

[0123] The CPU 806 can implement power management capabilities including one or more of the following features: individual hardware blocks can be automatically clock-gated when idle to save dynamic power; each core clock can be gated when the core is not actively executing instructions due to execution of WFI / WFE instructions; each core can be independently power-gated; each core cluster can be independently clock-gated when all cores are clock-gated or power-gated; and / or each core cluster can be independently power-gated when all cores are power-gated. The CPU 806 can further implement enhanced algorithms for managing power states in which the allowed power states and desired wake-up times are specified and the hardware / microcode determines the optimal power state for the cores, clusters, and CCPLEX to enter. The processing cores can support a simplified power state entry sequence in software, with the work being offloaded to microcode.

[0124] ​GPU 808 can include an integrated GPU (alternatively referred to herein as an “iGPU”). GPU 808 can be programmable and efficient for parallel workloads. In some examples, GPU 808 can use an enhanced tensor instruction set. GPU 808 can include one or more streaming microprocessors, where each streaming microprocessor can include an LI cache (e.g., an LI cache having at least 96 KB of storage capacity), and two or more of the streaming microprocessors can share an L2 cache (e.g., an L2 cache having 512 KB of storage capacity). In some embodiments, GPU 808 can include at least eight streaming microprocessors. GPU 808 can use a compute application programming interface (API). In addition, GPU 808 can use one or more parallel computing platforms and / or programming models (e.g., NVIDIA’s CUDA).

[0125] In the case of automotive and embedded uses, GPU 808 can be power-optimized for best performance. For example, GPU 808 can be fabricated on a fin field effect transistor (FinFET). However, this is not intended to be limiting, and GPU 808 can be fabricated using other semiconductor fabrication processes. Each streaming microprocessor can incorporate several mixed-precision processing cores divided into multiple blocks. For example, and without limitation, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA Tensor cores for deep learning matrix arithmetic, an L0 instruction cache, a thread warp scheduler, a dispatch unit, and / or a 64 KB register file. In addition, the streaming microprocessor can include independent parallel integer and floating point data paths to exploit the mix of computation and address computation for efficient execution of workloads. The streaming microprocessor can include independent thread scheduling capabilities to allow for more fine-grained synchronization and cooperation between parallel threads. The streaming microprocessor can include a combined LI data cache and shared memory unit to improve performance while simplifying programming.

[0126] GPU 808 can include a high bandwidth memory (HBM) and / or a 16 GB HBM2 memory subsystem that provides approximately 900 GB / s of peak memory bandwidth in some examples. In some examples, in addition to or alternatively from HBM memory, a synchronous graphics random access memory (SGRAM) can be used, such as a fifth generation graphics double data rate synchronous random access memory (GDDR5).

[0127] GPU 808 can include a unified memory technology that includes access counters to allow memory pages to be migrated more precisely to the processors that access them most frequently, improving efficiency of memory ranges shared between processors. In some examples, address translation services (ATS) support can be used to allow GPU 808 to access CPU 806 page tables directly. In such examples, when a GPU 808 memory management unit (MMU) experiences a miss, an address translation request can be transmitted to CPU 806. In response, CPU 806 can look up a virtual-to-physical mapping for the address in its page tables and transmit the translation back to GPU 808. In this way, the unified memory technology can allow a single unified virtual address space for memory of both CPU 806 and GPU 808, simplifying GPU 808 programming and porting applications to GPU 808.

[0128] Further, GPU 808 can include access counters that can track how frequently GPU 808 accesses other processors' memory. The access counters can help ensure that memory pages are migrated to the physical memory of the processor that accesses these pages most frequently.

[0129] SoC 804 can include any number of caches 812, including those described herein. For example, caches 812 can include an L3 cache available to both CPU 806 and GPU 808 (e.g., connected to both CPU 806 and GPU 808). Caches 812 can include a write-back cache that can track the state of a line, for example, by using a cache coherency protocol (e.g., MEI, MESI, MSI, etc.). Depending on the embodiment, the L3 cache can include 4 MB or more, although smaller cache sizes can also be used.

[0130] SoC 804 can include an arithmetic logic unit (ALU) that can be utilized in processing to perform any of a variety of tasks or operations with respect to vehicle 800, such as processing a DNN. Further, SoC 804 can include a floating point unit (FPU) (or other mathematical co-processor or digital co-processor type) for performing mathematical operations within the system. For example, SoC 104 can include one or more FPUs integrated as execution units within CPU 806 and / or GPU 808.

[0131] The SoC 804 can include one or more accelerators 814 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, the SoC 804 can include a hardware accelerator cluster that can include optimized hardware accelerators and / or a large on-chip memory. The large on-chip memory (e.g., 4 MB SRAM) can enable the hardware accelerator cluster to accelerate neural networks and other computations. The hardware accelerator cluster can be used to supplement the GPU 808 and offload some of the tasks of the GPU 808 (e.g., freeing up more cycles of the GPU 808 for performing other tasks). As one example, the accelerators 814 can be used for targeted workloads (e.g., perception, convolutional neural networks (CNNs), etc.) that are stable enough to be easily controlled for acceleration. As used herein, the term “CNN” can include all types of CNNs, including region-based or region with convolutional neural networks (RCNNs) and fast RCNNs (e.g., for object detection).

[0132] The accelerators 814 (e.g., hardware accelerator cluster) can include a deep learning accelerator (DLA). The DLA can include one or more tensor processing units (TPUs) that can be configured to provide an additional 100 billion operations per second for deep learning applications and inferencing. The TPU can be an accelerator that is configured to perform and optimized for performing image processing functions (e.g., for CNNs, RCNNs, etc.). The DLA can be further optimized for a specific set of neural network types and floating point operations and inferencing. The design of the DLA can provide higher performance per mm than a general purpose GPU and far exceeds the performance of a CPU. The TPU can perform several functions, including single instance convolution functions, support for INT8, INT16, and FP16 data types for both features and weights, for example, and post-processor functions.

[0133] The DLA can perform neural networks, especially CNNs, on processed or unprocessed data for any of a wide variety of functions, such as and not limited to: CNNs for object recognition and detection using data from a camera sensor; CNNs for distance estimation using data from a camera sensor; CNNs for emergency vehicle detection and identification and detection using data from a microphone; CNNs for face recognition and vehicle owner identification using data from a camera sensor; and / or CNNs for safety and / or safety related events.

[0134] The DLA can perform any of the functions of the GPU 808 and, by using an inferencing accelerator, for example, a designer can target the DLA or the GPU 808 for any function. For example, the designer can focus the processing and floating point operations of a CNN on the DLA and leave other functions to the GPU 808 and / or other accelerators 814.

[0135] Accelerator 814 (e.g., a cluster of hardware accelerators) can include a programmable vision accelerator (PVA), which can be alternatively referred to herein as a computer vision accelerator. The PVA can be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. The PVA can provide a balance between performance and flexibility. For example, each PVA can include, for example and without limitation, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors.

[0136] The RISC cores can interact with image sensors (e.g., image sensors of any of the cameras described herein), image signal processors, and / or the like. Each of the RISC cores can include any number of memories. Depending on the embodiment, the RISC cores can use any of several protocols. In some examples, the RISC cores can execute a real-time operating system (RTOS). The RISC cores can be implemented using one or more integrated circuit devices, application specific integrated circuits (ASICs), and / or memory devices. For example, the RISC cores can include an instruction cache and / or a tightly coupled RAM.

[0137] The DMA can enable components of the PVA to access system memory independently of the CPU 806. The DMA can support any number of features to provide optimizations to the PVA, including, but not limited to, supporting multi-dimensional addressing and / or circular addressing. In some examples, the DMA can support addressing up to six or more dimensions, which can include block width, block height, block depth, horizontal block stride, vertical block stride, and / or depth stride.

[0138] The vector processors can be programmable processors that can be designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing capabilities. In some examples, the PVA can include a PVA core and two vector processing subsystem partitions. The PVA core can include a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. The vector processing subsystems can operate as the main processing engines of the PVA and can include a vector processing unit (VPU), an instruction cache, and / or a vector memory (e.g., VMEM). The VPU core can include a digital signal processor, such as, for example, a single instruction multiple data (SIMD), very long instruction word (VLIW) digital signal processor. The combination of SIMD and VLIW can enhance throughput and rate.

[0139] Each of the vector processors can include an instruction cache and can be coupled to a dedicated memory. As a result, in some examples, each of the vector processors can be configured to execute independently of the other vector processors. In other examples, the vector processors included in a particular PVA can be configured to employ data parallelization. For example, in some embodiments, multiple vector processors included in a single PVA can execute the same computer vision algorithm, but on different regions of an image. In other examples, the vector processors included in a particular PVA can execute different computer vision algorithms on the same image simultaneously, or even different algorithms on a sequence of images or portions of an image. Any number of PVAs can be included in the hardware accelerator cluster, and any number of vector processors can be included in each of the PVAs, among other things. Furthermore, the PVAs can include additional error-correcting code (ECC) memory to enhance overall system security.

[0140] The accelerator 814 (e.g., hardware accelerator cluster) can include an on-chip computer vision network and SRAM to provide high bandwidth, low latency SRAM for the accelerator 814. In some examples, the on-chip memory can include at least 4 MB of SRAM composed of, for example and without limitation, eight field-programmable memory blocks, which can be accessed by both the PVA and the DLA. Each pair of memory blocks can include an advanced peripheral bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory can be used. The PVA and the DLA can access the memory via a backbone that provides high-speed memory access to the PVA and the DLA. The backbone can include an on-chip computer vision network that interconnects the PVA and the DLA to the memory, for example using an APB.

[0141] The on-chip computer vision network can include an interface that determines that both the PVA and the DLA provide ready and valid signals before transmitting any control signals / addresses / data. Such an interface can provide separate phases and separate channels for transmitting control signals / addresses / data, as well as burst communications for continuous data transmission. This type of interface can comply with ISO 26262 or IEC 61508 standards, but other standards and protocols can also be used.

[0142] In some examples, the SoC 804 can include a real-time ray tracing hardware accelerator, such as described in U.S. Patent Application No. 16 / 101,232, filed August 10, 2018. The real-time ray tracing hardware accelerator can be used to quickly and efficiently determine locations and extents of objects (e.g., within a world model) in order to generate real-time visualizations simulations for RADAR signal interpretation, for sound propagation synthesis and / or analysis, for SONAR system simulation, for general wave propagation simulation, for comparison with LIDAR data for purposes of localization and / or other functions, and / or for other uses. In some embodiments, one or more tree traversal units (TTUs) can be used to perform one or more ray tracing related operations.

[0143] The accelerator 814 (e.g., a hardware accelerator cluster) has a wide range of uses for autonomous driving. The PVA can be a programmable vision accelerator that can be used for key processing stages in ADAS and autonomous vehicles. The capabilities of the PVA are a good match for algorithm domains that require predictable processing, low power, and low latency. In other words, the PVA performs well on semi-dense or dense regular computations, and even on small data sets that require predictable runtimes with low latency and low power. Thus, in the context of a platform for autonomous vehicles, the PVA is designed to run classical computer vision algorithms because they are effective at object detection and integer math operations.

[0144] For example, according to one embodiment of the technology, the PVA is used to perform computer stereo vision. In some examples, a semi-global matching based algorithm can be used, although this is not intended to be limiting. Many applications for level 3-5 autonomous driving require instant motion estimation / stereo matching (e.g., structure from motion, pedestrian recognition, lane detection, etc.). The PVA can perform computer stereo vision functions on input from two monocular cameras.

[0145] In some examples, the PVA can be used to perform dense optical flow. Raw RADAR data is processed according to a process (e.g., using a 4D fast Fourier transform) to provide processed RADAR. In other examples, the PVA is used for time-of-flight depth processing, such as by processing raw time-of-flight data to provide processed time-of-flight data.

[0146] The DLA can be used to run any type of network to enhance control and driving safety, including, for example, a neural network that outputs a confidence metric for each object detection. Such a confidence value can be interpreted as a probability, or as providing a relative "weight" for each detection compared to other detections. The confidence value enables the system to make further decisions about which detections should be considered true positive detections and not false positive detections. For example, the system can set a threshold for confidence, and only consider detections that exceed the threshold as true positive detections. In an automatic emergency braking (AEB) system, false positive detections would cause the vehicle to automatically perform an emergency brake, which is obviously undesirable. Thus, only the most confident detections should be considered a trigger for AEB. The DLA can run a neural network for regression of a confidence value. The neural network can take as its input at least some subset of parameters, such as a bounding box dimension, a ground plane estimate obtained (e.g., from another subsystem), inertial measurement unit (IMU) sensor 866 outputs related to vehicle 800 orientation, distance, 3D position estimates of objects obtained from the neural network and / or other sensors (e.g., LIDAR sensor 864 or RADAR sensor 860), etc.

[0147] SoC 804 can include one or more data stores 816 (e.g., memory). Data stores 816 can be on-chip memory of SoC 804, which can store neural networks to be executed on the GPU and / or DLA. In some examples, for redundancy and safety, data stores 816 can be large enough in capacity to store multiple instances of a neural network. Data stores 812 can include L2 or L3 cache 812. References to data stores 816 can include references to memory associated with PVAs, DLAs, and / or other accelerators 814 as described herein.

[0148] SoC 804 can include one or more processors 810 (e.g., embedded processors). The processors 810 can include a boot and power management processor, which can be a specialized processor and subsystem for handling boot power and management functions, as well as security implementation. The boot and power management processor can be part of the SoC 804 boot sequence and can provide run-time power management services. The boot power and management processor can provide clock and voltage programming, auxiliary system low power state transitions, SoC 804 thermal and temperature sensor management, and / or SoC 804 power state management. Each temperature sensor can be implemented as a ring oscillator whose output frequency is proportional to temperature, and the SoC 804 can use the ring oscillator to detect the temperature of the CPU 806, GPU 808, and / or accelerator 814. If it is determined that the temperature exceeds a threshold, the boot and power management processor can enter a temperature fault routine and place the SoC 804 in a lower power state and / or place the vehicle 800 in a driver safe park mode (e.g., safely park the vehicle 800).

[0149] The processors 810 can also include a set of embedded processors that can be used as an audio processing engine. The audio processing engine can be an audio subsystem that allows for full hardware support for multi-channel audio over multiple interfaces, as well as a range of extensive and flexible audio I / O interfaces. In some examples, the audio processing engine is a specialized processor core with a digital signal processor with dedicated RAM.

[0150] The processors 810 can also include an always-on processor engine, which can provide the necessary hardware features to support low-power sensor management and wake-up use cases. The always-on processor engine can include a processor core, tightly coupled RAM, supporting peripherals (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0151] The processors 810 can also include a security cluster engine, which includes a specialized processor subsystem that handles security management for automotive applications. The security cluster engine can include two or more processor cores, tightly coupled RAM, supporting peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In a secure mode, the two or more cores can operate in a lockstep mode and act as a single core with comparison logic that detects any differences between their operations.

[0152] The processors 810 can also include a real-time camera engine, which can include a specialized processor subsystem for handling real-time camera management.

[0153] The processor 810 can further include a high dynamic range signal processor, which can include an image signal processor, which is a hardware engine that is part of the camera processing pipeline.

[0154] The processor 810 can include a video image compositor, which can be a processing block (e.g., implemented on a microprocessor), that implements video post-processing functions needed by the video playback application to produce the final image for the player window. The video image compositor can perform lens distortion correction on the wide-angle camera 870, surround camera 874, and / or on the cab-in monitor camera sensors. The cab-in monitor camera sensors are preferably monitored by a neural network running on another instance of the advanced SoC, configured to recognize cab-in events and respond accordingly. The cab-in system can perform lip reading to activate mobile phone services and place a call, dictate an email, change the vehicle destination, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. Certain functions are only available to the driver when the vehicle is operating in autonomous mode, and are disabled otherwise.

[0155] The video image compositor can include enhanced temporal noise reduction for spatial and temporal noise reduction. For example, where motion is present in the video, the noise reduction appropriately weights the spatial information, reducing the weight of information provided by neighboring frames. Where the image or portions of the image do not include motion, the temporal noise reduction performed by the video image compositor can use information from previous images to reduce noise in the current image.

[0156] The video image compositor can also be configured to perform stereo correction on input stereo lens frames. The video image compositor can further be used for user interface composition when the operating system desktop is in use and the GPU 808 does not need to continuously render new surfaces. Even when the GPU 808 is powered on and active, doing 3D rendering, the video image compositor can be used to offload the GPU 808 to improve performance and responsiveness.

[0157] The SoC 804 can further include a Mobile Industry Processor Interface (MIPI) camera serial interface for receiving video and input from cameras, a high-speed interface, and / or a video input block that can be used for camera and related pixel input functions. The SoC 804 can further include an input / output controller that can be controlled by software and can be used to receive I / O signals that are not committed to a particular role.

[0158] The SoC 804 can also include a wide range of peripheral device interfaces to enable communication with peripherals, audio codecs, power management, and / or other devices. The SoC 804 can be used to process data from cameras (connected over Gigabit Multimedia Serial Link and Ethernet), sensors (e.g., LIDAR sensors 864, RADAR sensors 860, etc. that can be connected over Ethernet), data from the bus 802 (e.g., speed of the vehicle 800, steering wheel position, etc.), data from GNSS sensors 858 (connected over Ethernet or CAN bus). The SoC 804 can also include dedicated high-performance mass storage controllers, which can include their own DMA engines, and which can be used to free up the CPU 806 from routine data management tasks.

[0159] The SoC 804 can be an end-to-end platform with a flexible architecture that spans automation levels 3-5, providing an integrated functional safety architecture for a platform that leverages and efficiently uses computer vision and ADAS technology to achieve diversity and redundancy, along with deep learning tools. The SoC 804 can be faster, more reliable, and even more energy and space efficient than conventional systems. For example, the accelerators 814, when combined with the CPU 806, GPU 808, and data storage 816, can provide a fast and efficient platform for level 3-5 autonomous vehicles.

[0160] The technology thus provides capabilities and functionality that cannot be achieved by conventional systems. For example, computer vision algorithms can be executed on CPUs that can be configured using high-level programming languages such as the C programming language to perform a wide variety of processing algorithms across a wide variety of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to, for example, execution time and power consumption. In particular, many CPUs cannot execute complex object detection algorithms in real time, which is a requirement for on-board ADAS applications and a requirement for practical level 3-5 autonomous vehicles.

[0161] In contrast to conventional systems, by providing a CPU complex, a GPU complex, and a cluster of hardware accelerators, the technology described herein allows multiple neural networks to be executed simultaneously and / or sequentially, and the results to be combined together to achieve level 3-5 autonomous driving functionality. For example, a CNN executed on a DLA or dGPU (e.g., GPU 820) can include text and word recognition, allowing a supercomputer to read and understand traffic signs, including signs for which a neural network has not been specifically trained. The DLA can further include a neural network that is able to recognize, interpret, and provide a semantic understanding of the sign, and pass that semantic understanding to a path planning module running on the CPU complex.

[0162] As another example, multiple neural networks can be run simultaneously as required for level 3, 4, or 5 driving. For example, a warning sign consisting of the words "Caution: flashing lights indicate icy conditions" along with electric lights can be interpreted by several neural networks independently or collectively. The sign itself can be recognized by a first deployed neural network (e.g., a trained neural network) as a traffic sign, the text "flashing lights indicate icy conditions" can be interpreted by a second deployed neural network that informs the vehicle's path planning software (preferably executing on the CPU complex) that icy conditions exist when flashing lights are detected. The flashing lights can be recognized by operating a third deployed neural network over multiple frames that informs the vehicle's path planning software of the presence (or absence) of flashing lights. All three neural networks can be run simultaneously, for example, within the DLA and / or on the GPU 808.

[0163] In some examples, a CNN for face recognition and owner recognition can use data from the camera sensor to recognize the presence of an authorized driver and / or owner of the vehicle 800. A processing engine always on the sensor can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and in a safe mode, disable the vehicle when the owner leaves the vehicle. In this way, the SoC 804 provides security against theft and / or carjacking.

[0164] In another example, a CNN for emergency vehicle detection and recognition can use data from the microphones 896 to detect and recognize emergency vehicle sirens. In contrast to conventional systems that detect sirens using a general classifier and manually extract features, the SoC 804 uses a CNN to classify ambient and urban sounds as well as to classify visual data. In a preferred embodiment, a CNN running on the DLA is trained to recognize the relative closing speed of an emergency vehicle (e.g., by using the Doppler effect). The CNN can also be trained to recognize emergency vehicles specific to the local area in which the vehicle is operating as recognized by the GNSS sensor 858. Thus, for example, when operating in Europe, the CNN will seek to detect European sirens, and when in the United States, the CNN will seek to recognize sirens that are only North American. Once an emergency vehicle is detected, a control program can be used to execute an emergency vehicle safety routine, slow the vehicle down, pull over to the side of the road, stop the vehicle, and / or idle the vehicle until the emergency vehicle passes, with the assistance of the ultrasonic sensors 862.

[0165] The vehicle can include a CPU 818 (e.g., a discrete CPU or dCPU) that can be coupled to the SoC 804 via a high-speed interconnect (e.g., PCIe). The CPU 818 can include, for example, an X86 processor. The CPU 818 can be used to perform any of a wide variety of functions, including, for example, arbitrating potentially inconsistent results between ADAS sensors and the SoC 804, and / or monitoring the status and health of the controller 836 and / or infotainment SoC 830.

[0166] The vehicle 800 can include a GPU 820 (e.g., a discrete GPU or dGPU) that can be coupled to the SoC 804 via a high-speed interconnect (e.g., NVIDIA’s NVLINK). The GPU 820 can provide additional artificial intelligence functionality, for example, by executing redundant and / or different neural networks, and can be used to train and / or update neural networks based at least in part on input (e.g., sensor data) from sensors of the vehicle 800.

[0167] The vehicle 800 can also include a network interface 824 that can include one or more wireless antennas 826 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). The network interface 824 can be used to enable wireless connections over the Internet with a cloud (e.g., with a server 878 and / or other network devices), with other vehicles, and / or with computing devices (e.g., client devices of passengers). For communication with other vehicles, a direct link can be established between the two vehicles, and / or an indirect link can be established (e.g., across a network and through the Internet). The direct link can be provided using a car-to-car communication link. The car-to-car communication link can provide the vehicle 800 with information about vehicles that are approaching the vehicle 800 (e.g., vehicles in front of, to the side of, and / or behind the vehicle 800). This functionality can be part of a cooperative adaptive cruise control functionality of the vehicle 800.

[0168] The network interface 824 can include a SoC that provides modulation and demodulation functionality and enables the controller 836 to communicate over a wireless network. The network interface 824 can include a radio frequency front end for up-conversion from baseband to radio frequency and down-conversion from radio frequency to baseband. The frequency conversion can be performed through well-known processes, and / or can be performed using a super-heterodyne process. In some examples, the radio frequency front end functionality can be provided by a separate chip. The network interface can include wireless functionality for communication over LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0169] The vehicle 800 can also include a data store 828, which can include off-chip (e.g., off-SoC 804) storage. The data store 828 can include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash memory, hard disks, and / or other components and / or devices that can store data for at least one bit.

[0170] The vehicle 800 can also include a GNSS sensor 858. The GNSS sensor 858 (e.g., GPS, assisted GPS sensor, differential GPS (DGPS) sensor, etc.) is used to assist in mapping, perception, occupancy grid generation, and / or path planning functions. Any number of GNSS sensors 858 can be used, including, for example and without limitation, a GPS using a USB connector with an Ethernet-to-serial (RS-232) bridge.

[0171] The vehicle 800 can also include a RADAR sensor 860. The RADAR sensor 860 can be used by the vehicle 800 for long-range vehicle detection, even in darkness and / or adverse weather conditions. The RADAR functional safety level can be ASIL B. The RADAR sensor 860 can use the CAN and / or bus 802 (e.g., to transmit data generated by the RADAR sensor 860) for control as well as access to object tracking data, in some examples, Ethernet for access to raw data. A wide variety of RADAR sensor types can be used. For example and without limitation, the RADAR sensor 860 can be suitable for front, rear, and side RADAR use. In some examples, a pulsed Doppler RADAR sensor is used.

[0172] The RADAR sensor 860 can include different configurations, such as long-range with narrow field of view, short-range with wide field of view, short-range side coverage, and so on. In some examples, long-range RADAR can be used for adaptive cruise control functionality. Long-range RADAR systems can provide a wide field of view (e.g., 250 m range) implemented through two or more independent scans. The RADAR sensor 860 can help distinguish between static and moving objects, and can be used by the ADAS system for emergency brake assist and forward collision warning. Long-range RADAR sensors can include a single-station multi-mode RADAR with multiple (e.g., six or more) fixed RADAR antennas, as well as high-speed CAN and FlexRay interfaces. In examples with six antennas, the central four antennas can create focused beam patterns designed to record the surroundings of the vehicle 800 at higher speed with minimal traffic interference from adjacent lanes. The other two antennas can extend the field of view, making it possible to quickly detect vehicles entering or leaving the lane of the vehicle 800. In some aspects, the sensor data 102 can include data generated by one or more RADAR sensors 860.

[0173] As one example, a mid-range RADAR system can include a range of up to 860 m (front) or 80 m (rear) and a field of view of up to 42 degrees (front) or 850 degrees (rear). A short-range RADAR system can include, but is not limited to, RADAR sensors designed to be mounted at both ends of the rear bumper. When mounted at both ends of the rear bumper, such a RADAR sensor system can create two beams that continuously monitor the rear and the blind spot next to the vehicle.

[0174] A short-range RADAR system can be used in an ADAS system for blind spot detection and / or lane change assist.

[0175] The vehicle 800 can also include ultrasonic sensors 862. The ultrasonic sensors 862, which can be placed on the front, rear, and / or sides of the vehicle 800, can be used for parking assist and / or to create and update an occupancy grid. A wide variety of ultrasonic sensors 862 can be used, and different ultrasonic sensors 862 can be used for different detection ranges (e.g., 2.5 m, 4 m). The ultrasonic sensors 862 can operate at an ASIL B functional safety level. In some aspects, the sensor data 102 can include data generated by one or more ultrasonic sensors 862.

[0176] The vehicle 800 can include LIDAR sensors 864. The LIDAR sensors 864 can be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LIDAR sensors 864 can be at an ASIL B functional safety level. In some examples, the vehicle 800 can include multiple LIDAR sensors 864 (e.g., two, four, six, etc.) that can use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0177] In some examples, the LIDAR sensors 864 can be capable of providing a list of objects and their distances for a 360-degree field of view. A commercially available LIDAR sensor 864 can have, for example, an advertised range of approximately 800 m, a precision of 2 cm - 3 cm, and support for an 800 Mbps Ethernet connection. In some examples, one or more flush-mounted LIDAR sensors 864 can be used. In such examples, the LIDAR sensors 864 can be implemented as small devices that can be embedded into the front, rear, sides, and / or corners of the vehicle 800. In such examples, the LIDAR sensors 864 can provide a field of view of up to 120 degrees horizontal and 35 degrees vertical, with a range of 200 m, even for low reflectivity objects. Front-mounted LIDAR sensors 864 can be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0178] In some examples, LIDAR technology such as 3D Flash LIDAR can also be used. 3D Flash LIDAR uses a flash of laser light as a source of emission to illuminate the vehicle’s surroundings up to about 200 m. The flash LIDAR unit includes a receptor that records the laser pulse transmission time and reflected light on each pixel, which in turn corresponds to the range from the vehicle to the object. Flash LIDAR can allow for the generation of highly accurate and distortion-free images of the surroundings with each laser flash. In some examples, four flash LIDAR sensors can be deployed, one on each side of the vehicle 800. Available 3D flash LIDAR systems include solid-state 3D staring array LIDAR cameras (e.g., non-scanning LIDAR devices) that have no moving parts other than fans. The flash LIDAR device can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture the reflected laser light in the form of 3D range point clouds and co-registered intensity data. By using flash LIDAR, and because flash LIDAR is a solid-state device with no moving parts, the LIDAR sensor 864 can be less susceptible to motion blur, vibration, and / or jostling. In some aspects, the sensor data 102 can include data generated by one or more LIDAR sensors 864.

[0179] The vehicle can also include an IMU sensor 866. In some examples, the IMU sensor 866 can be located at the center of the rear axle of the vehicle 800. The IMU sensor 866 can include, for example and without limitation, an accelerometer, a magnetometer, a gyroscope, a magnetic compass, and / or other sensor types. In some examples, such as in six-axis applications, the IMU sensor 866 can include an accelerometer and a gyroscope, while in nine-axis applications, the IMU sensor 866 can include an accelerometer, a gyroscope, and a magnetometer.

[0180] In some embodiments, the IMU sensor 866 can be implemented as a microelectromechanical systems (MEMS) inertial navigation system (INS) that combines a microelectromechanical system (MEMS) inertial sensor, a high-sensitivity GPS receiver, and an advanced Kalman filter algorithm to provide estimates of position, velocity, and attitude. As such, in some examples, the IMU sensor 866 can enable the vehicle 800 to estimate heading without input from a magnetic sensor by directly observing the change in velocity from GPS to the IMU sensor 866 and correlating it. In some examples, the IMU sensor 866 and the GNSS sensor 858 can be combined into a single integrated unit.

[0181] The vehicle can include microphones 896 placed in and / or around the vehicle 800. The microphones 896 can be used for emergency vehicle detection and identification, among other things.

[0182] The vehicle can also include any number of camera types, including stereo cameras 868, wide-view cameras 870, infrared cameras 872, surround cameras 874, long and / or mid-range cameras 898, and / or other camera types. These cameras can be used to capture image data around the entire periphery of the vehicle 800. The types of cameras used depend on the embodiment and requirements of the vehicle 800, and any combination of camera types can be used to provide the necessary coverage around the vehicle 800. Further, the number of cameras can vary depending on the embodiment. For example, the vehicle can include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. As one example and not by way of limitation, the cameras can support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet. Each of the cameras is described in more detail herein with respect to Figure 8A and Figure 8B are described in more detail.

[0183] The vehicle 800 can also include vibration sensors 842. The vibration sensors 842 can measure vibrations of components of the vehicle, such as axles. For example, changes in vibration can indicate changes in the road surface. In another example, when two or more vibration sensors 842 are used, differences between the vibrations can be used to determine the friction or slip of the road surface (e.g., when there is a difference in vibration between a power driven axle and a free spinning axle).

[0184] The vehicle 800 can include an ADAS system 838. In some examples, the ADAS system 838 can include a SoC. The ADAS system 838 can include adaptive / automatic / autonomous cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keep assist (LKA), blind spot warning (BSW), rear cross-traffic warning (RCTW), collision warning system (CWS), lane centering (LC), and / or other features and functionality.

[0185] An ACC system can use RADAR sensors 860, LIDAR sensors 864, and / or cameras. The ACC system can include longitudinal ACC and / or lateral ACC. Longitudinal ACC monitors and controls the distance to the vehicle immediately ahead of the vehicle 800 and automatically adjusts the vehicle speed to maintain a safe distance from the vehicle ahead. Lateral ACC performs distance keeping and, if necessary, suggests a lane change for the vehicle 800. Lateral ACC is related to other ADAS applications such as LCA and CWS.

[0186] CACC uses information from other vehicles, which can be received from other vehicles via a wireless link via the network interface 824 and / or wireless antenna 826 or indirectly through a network connection, such as through the Internet. Direct links can be provided by vehicle-to-vehicle (V2V) communication links, while indirect links can be infrastructure-to-vehicle (I2V) communication links. Generally, V2V communication concepts provide information about the immediately preceding vehicles, such as vehicles immediately ahead of and in the same lane as the vehicle 800, while I2V communication concepts provide information about traffic further ahead. A CACC system can include either or both of I2V and V2V information sources. Given information about vehicles ahead of the vehicle 800, CACC can be more reliable, and it has the potential to improve traffic flow and reduce road congestion.

[0187] FCW systems are designed to alert the driver to a hazard so that the driver can take corrective action. FCW systems use a front-facing camera and / or RADAR sensor 860 coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as displays, speakers, and / or vibrating components. FCW systems can provide warnings in the form of, for example, sound, visual warnings, vibrations, and / or quick brake pulses.

[0188] AEB systems detect an impending forward collision with another vehicle or other object and can automatically apply the brakes if the driver does not take corrective action within specified time or distance parameters. AEB systems can use a front-facing camera and / or RADAR sensor 860 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. When an AEB system detects a hazard, it typically first alerts the driver to take corrective action to avoid a collision, and if the driver does not take corrective action, the AEB system can automatically apply the brakes in an effort to prevent or at least mitigate the effects of a predicted collision. AEB systems can include technologies such as dynamic brake support and / or crash imminent braking.

[0189] LDW systems provide visual, audible, and / or tactile warnings, such as steering wheel or seat vibrations, to alert the driver when the vehicle 800 is crossing lane markers. The LDW system is not activated when the driver indicates an intentional lane departure by activating a turn signal. LDW systems can use a front-side facing camera coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as displays, speakers, and / or vibrating components.

[0190] An LKA system is a variation of the LDW system. If the vehicle 800 begins to leave the lane, the LKA system provides a steering input or brake to correct the vehicle 800.

[0191] A BSW system detects and warns the driver of vehicles in the car's blind spot. The BSW system can provide visual, audible, and / or tactile alerts to indicate that merging or changing lanes is unsafe. The system can provide additional warnings when the driver uses a turn signal. The BSW system can use rear side-facing cameras and / or RADAR sensors 860 coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as a display, speaker, and / or vibrating component.

[0192] A RCTW system can provide visual, audible, and / or tactile notifications when objects are detected outside the range of the rear-facing camera while the vehicle 800 is backing up. Some RCTW systems include AEB to ensure that vehicle brakes are applied to avoid a collision. The RCTW system can use one or more rear-facing RADAR sensors 860 coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as a display, speaker, and / or vibrating component.

[0193] Conventional ADAS systems can be prone to false positive results, which can annoy and distract the driver, but typically are not catastrophic because the ADAS system alerts the driver and allows the driver to decide whether the safety condition is truly present and act accordingly. However, in an autonomous vehicle 800, in the case of conflicting results, the vehicle 800 itself must decide whether to heed the results from the primary computer or the secondary computer (e.g., the first controller 836 or the second controller 836). For example, in some embodiments, the ADAS system 838 can be a secondary and / or auxiliary computer for providing perception information to a backup computer plausibility module. The backup computer plausibility monitor can run redundant diverse software on hardware components to detect faults in perception and dynamic driving tasks. The output from the ADAS system 838 can be provided to a supervisory MCU. If the outputs from the primary computer and the secondary computer conflict, the supervisory MCU must determine how to reconcile the conflict to ensure safe operation.

[0194] In some examples, the host computer can be configured to provide a confidence score to the supervisory MCU indicating the host computer's confidence in the selected result. If the confidence score exceeds a threshold, then the supervisory MCU can follow the host computer's direction, regardless of whether the secondary computer provides conflicting or inconsistent results. In the event that the confidence score does not satisfy the threshold and in the event that the host computer and the secondary computer indicate different results (e.g., a conflict), the supervisory MCU can arbitrate between the computers to determine the appropriate result.

[0195] The supervisory MCU can be configured to run a neural network that is trained and configured to determine conditions under which the secondary computer provides false alarms based at least in part on the output from the host computer and the secondary computer. Thus, the neural network in the supervisory MCU can learn when the output of the secondary computer can be trusted and when it cannot. For example, when the secondary computer is a RADAR-based FCW system, the neural network in the supervisory MCU can learn when the FCW system is identifying metal objects that are not in fact dangerous, such as drain grates or manhole covers that trigger false alarms. Similarly, when the secondary computer is a camera-based LDW system, the neural network in the supervisory MCU can learn to disregard the LDW when a cyclist or pedestrian is present and lane departure is in fact the safest strategy. In embodiments that include a neural network running on the supervisory MCU, the supervisory MCU can include at least one of a DLA or a GPU suitable for running a neural network with associated memory. In preferred embodiments, the supervisory MCU can include and / or be included as a component of the SoC 804.

[0196] In other examples, the ADAS system 838 can include a secondary computer that performs ADAS functions using traditional computer vision rules. In this way, the secondary computer can use classic computer vision rules (if-then), and the presence of a neural network in the supervisory MCU can improve reliability, safety, and performance. For example, the diverse implementation and intentional non-identity make the overall system more fault-tolerant, especially with respect to faults caused by software (or software-hardware interface) functions. For example, if there is a software bug or error in the software running on the host computer and the non-identical software code running on the secondary computer provides the same overall result, then the supervisory MCU can be more confident that the overall result is correct and that the bug in the software or hardware on the host computer did not cause a substantial error.

[0197] In some examples, the output of the ADAS system 838 can be fed to a perception block of the host computer and / or a dynamic driving task block of the host computer. For example, if the ADAS system 838 indicates a forward collision warning due to an object immediately ahead, the perception block can use this information in identifying the object. In other examples, the secondary computer can have its own neural network that is trained and thus reduces the risk of false positives as described herein.

[0198] The vehicle 800 can also include an infotainment SoC 830 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as a SoC, the infotainment system can not be a SoC and can include two or more discrete components. The infotainment SoC 830 can include a combination of hardware and software that can be used to provide audio (e.g., music, a personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., TV, movies, streaming media, etc.), telephony (e.g., hands-free calling), network connectivity (e.g., LTE, WiFi, etc.), and / or information services (e.g., a navigation system, a park assist, a telematics device, a radio data system, vehicle-related information such as fuel level, total distance covered, brake fluid level, oil level, doors open / closed, air filter information, etc.) to the vehicle 800. For example, the infotainment SoC 830 can include a radio, a disc player, a navigation system, a video player, USB and Bluetooth connectivity, an in-car computer, in-car entertainment, WiFi, steering wheel audio controls, hands-free voice controls, a heads-up display (HUD), the HMI display 834, a telematics device, a control panel (e.g., for controlling and / or interacting with various components, features, and / or systems), and / or other components. The infotainment SoC 830 can further be used to provide information (e.g., visual and / or audible) to a user of the vehicle, such as information from the ADAS system 838, autonomous driving information such as planned vehicle maneuvers, trajectories, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0199] The infotainment SoC 830 can include GPU functionality. The infotainment SoC 830 can communicate with other devices, systems, and / or components of the vehicle 800 over the bus 802 (e.g., a CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 830 can be coupled to a supervisory MCU such that, in the event of a failure of the host controller 836 (e.g., a primary and / or backup computer of the vehicle 800), the GPU of the infotainment system can perform some autonomous driving functions. In such examples, the infotainment SoC 830 can place the vehicle 800 in a driver safe park mode as described herein.

[0200] The vehicle 800 may also include an instrument cluster 832 (e.g., a digital instrument panel, an electronic instrument cluster, a digital instrument panel, etc.). The instrument cluster 832 may include a controller and / or a supercomputer (e.g., a separate controller or a supercomputer). The instrument cluster 832 may include a set of instruments, such as a speedometer, fuel level, oil pressure, a tachometer, an odometer, a turn indicator, a shift position indicator, a seat belt warning light, a parking brake warning light, an engine check light, airbag (SRS) system information, lighting controls, safety system controls, navigation information, etc. In some examples, information can be displayed and / or shared between the infotainment SoC 830 and the instrument cluster 832. In other words, the instrument cluster 832 may be included as part of the infotainment SoC 830, or vice versa.

[0201] Figure 8D For cloud-based servers and Figure 8A 880). Depending on the embodiment, each of the servers 878 can include any number of GPUs 884, CPUs 880, and / or PCIe switches. For example, each of the servers 878 can include eight, sixteen, thirty-two, and / or more GPUs 884.

[0202] The server 878 can receive image data from vehicles over the network 890 and representing images showing unexpected or changing road conditions such as a road work that recently started. The server 878 can transmit neural networks 892, updated neural networks 892, and / or map information 894, including information about traffic and road conditions, to vehicles over the network 890. Updates to the map information 894 can include updates to the HD map 822, e.g., information about construction sites, potholes, curves, floods, or other obstacles. In some examples, the neural networks 892, updated neural networks 892, and / or map information 894 can have been generated from experience using training performed at a data center (e.g., using the server 878 and / or other servers) and / or from data received from any number of vehicles in the environment.

[0203] The server 878 can be used to train machine learning models (e.g., neural networks) based on training data. The training data can be generated by vehicles and / or can be generated in simulations (e.g., using game engines). In some examples, the training data is labeled (e.g., in cases where the neural network benefits from supervised learning) and / or undergoes other pre-processing, while in other examples, the training data is not labeled and / or pre-processed (e.g., in cases where the neural network does not require supervised learning). The training can be performed according to any one or more categories of machine learning techniques, including but not limited to categories such as: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, federated learning, transfer learning, feature learning (including principal component and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including spare dictionary learning), rule-based machine learning, anomaly detection, and any variants or combinations thereof. Once the machine learning models are trained, the machine learning models can be used by vehicles (e.g., transmitted to vehicles over the network 890), and / or the machine learning models can be used by the server 878 to remotely monitor vehicles.

[0204] In some examples, the server 878 can receive data from vehicles and apply the data to the latest real-time neural networks for real-time intelligent inference. The server 878 can include deep learning supercomputers and / or specialized AI computers powered by GPUs 884, such as the DGX and DGX Station machines developed by NVIDIA. However, in some examples, the server 878 can include deep learning infrastructure of a data center that is powered by CPUs only.

[0205] The deep learning infrastructure of the server 878 can be capable of fast real-time inference, and can use this capability to assess and validate the health of the processors, software, and / or associated hardware in the vehicle 800. For example, the deep learning infrastructure can receive periodic updates from the vehicle 800, such as a sequence of images and / or objects located in the sequence of images that the vehicle 800 has located (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure can run its own neural network to identify the objects and compare them to the objects identified by the vehicle 800, and if the results do not match and the infrastructure concludes that the AI in the vehicle 800 is malfunctioning, then the server 878 can transmit a signal to the vehicle 800 instructing the fail-safe computer of the vehicle 800 to take control, notify the passengers, and complete a safe parking operation.

[0206] For inference, the server 878 can include GPUs 884 and one or more programmable inference accelerators (such as NVIDIA’s TensorRT). The combination of GPU-powered servers and inference-accelerated can make real-time response possible. In other examples, such as where performance is less important, CPU, FPGA, and other processor-powered servers can be used for inference.

[0207] Example Computing Device

[0208] Figure 9 is a block diagram of an example computing device 900 suitable for implementing some embodiments of the present disclosure. The computing device 900 can include an interconnection system 902 coupling the following devices: a memory 904, one or more central processing units (CPUs) 906, one or more graphics processing units (GPUs) 908, a communication interface 910, input / output (I / O) ports 912, I / O components 914, a power supply 916, one or more presentation components 918 (e.g., display(s)), and one or more logic units 920. In at least one embodiment, the computing device(s) 900 can include one or more virtual machines (VMs), and / or any component thereof can include a virtual component (e.g., a virtual hardware component). For non-limiting examples, one or more of the GPUs 908 can include one or more vGPUs, one or more of the CPUs 906 can include one or more vCPUs, and / or one or more of the logic units 920 can include one or more virtual logic units. As such, the computing device(s) 900 can include discrete components (e.g., a full GPU dedicated to the computing device 900), virtual components (e.g., a portion of a GPU dedicated to the computing device 900), or a combination thereof.

[0209] AlthoughFigure 9 The various blocks shown in FIG. 9 A are meant to indicate functional blocks of various components (e.g., a “means” for performing a described function) that can be combined, divided, re-arranged, and / or supplemented by other components. For example, the presentation component 918 (such as a display device) can be considered an I / O component 914 (e.g., if the display is a touchscreen). As another example, the CPU 906 and / or GPU 908 can include memory (e.g., memory 904 can represent a storage device in addition to the memory of the GPU 908, CPU 906, and / or other components). In other words, Figure 9 The computing device of FIG. 9 A is merely illustrative. No distinction is made between “workstation,” “server,” “laptop,” “desktop,” “tablet,” “client device,” “mobile device,” “handheld device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types as all are considered within the scope of the computing device of FIG. 9 A. Figure 9 The computing device of FIG. 9 A is merely illustrative. No distinction is made between “workstation,” “server,” “laptop,” “desktop,” “tablet,” “client device,” “mobile device,” “handheld device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types as all are considered within the scope of the computing device of FIG. 9 A.

[0210] The interconnect system 902 can represent one or more links or buses, such as an address bus, data bus, control bus, or a combination thereof. The interconnect system 902 can include one or more bus or link types, such as an Industry Standard Architecture (ISA) bus, an Extended Industry Standard Architecture (EISA) bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe) bus, and / or another type of bus or link. In some embodiments, there are direct connections between components. As an example, the CPU 906 can be directly connected to the memory 904. Further, the CPU 906 can be directly connected to the GPU 908. Where there are direct or point-to-point connections between components, the interconnect system 902 can include a PCIe link to perform the connection. In these examples, a PCI bus need not be included in the computing device 900.

[0211] The memory 904 can include any of a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by computing device 900. By way of example, and not limitation, computer-readable media can comprise computer storage media and communication media.

[0212] Computer storage media can include volatile and nonvolatile media and / or removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules, and / or other data types. For example, memory 904 can store computer readable instructions (e.g., representing program(s) and / or program element(s), such as an operating system). Computer storage media can include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by computing device 900. As used herein, computer storage media does not include signals per se.

[0213] Computer storage media can embody computer readable instructions, data structures, program modules, and / or other data types in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" can refer to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, computer storage media can include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of the any of the above should also be included within the scope of computer readable media.

[0214] CPUs 906 can each include one or more cores capable of handling a number of software threads concurrently (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.). CPUs 906 can include any type of processors, and can include different types of processors depending on the type of computing device 900 implemented (e.g., a mobile device having fewer cores for processors and a server having more cores for processors). For example, depending on the type of computing device 900, the processors can be Advanced RISC Machines (ARM) processors implemented using Reduced Instruction Set Computing (RISC) or x86 processors implemented using Complex Instruction Set Computing (CISC). Computing device 900 can include one or more CPUs 906 in addition to, or as an alternative to, one or more microprocessors or co-processors such as mathematical co-processors.

[0215] In addition to or in place of the CPU(s) 906, the GPU(s) 908 can be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 900 to perform one or more of the methods and / or processes described herein. One or more of the GPU(s) 908 can be integrated GPUs (e.g., with one or more of the CPU(s) 906) and / or one or more of the GPU(s) 908 can be discrete GPUs. In embodiments, one or more of the GPU(s) 908 can be a co-processor to one or more of the CPU(s) 906. The GPU(s) 908 can be used by the computing device 900 to render graphics (e.g., 3D graphics) or to perform general-purpose computing. For example, the GPU(s) 908 can be used for general-purpose computing on GPUs (GPGPU). The GPU(s) 908 can include hundreds or thousands of cores capable of handling hundreds or thousands of software threads concurrently. The GPU(s) 908 can generate pixel data for an output image in response to rendering commands (e.g., received from the CPU(s) 906 via a host interface). The GPU(s) 908 can include graphics memory (e.g., display memory) for storing pixel data or any other suitable data (e.g., GPGPU data). The display memory can be included as part of the memory 904. The GPU(s) 908 can include two or more GPUs operating in parallel (e.g., via a link). The link can connect the GPUs directly (e.g., using NVLINK) or can connect the GPUs through a switch (e.g., using an NVSwitch). When combined together, each GPU 908 can generate pixel data or GPGPU data for a different portion of an output or for a different output (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU can include its own memory or can share memory with other GPUs.

[0216] In addition to or in place of CPU(s) 906 and / or GPU(s) 908, logic units 920 can be configured to execute at least some of the computer-readable instructions to control one or more components of computing device 900 to perform one or more of the methods and / or processes described herein. In embodiments, CPU(s) 906, GPU(s) 908, and / or logic unit(s) 920 can perform any combination of the methods, processes, and / or portions thereof, discretely or jointly. One or more of logic units 920 can be part of and / or integrated in one or more of CPU(s) 906 and / or GPU(s) 908, and / or one or more of logic units 920 can be discrete components or otherwise external to CPU(s) 906 and / or GPU(s) 908. In embodiments, one or more of logic units 920 can be a co-processor of one or more of CPU(s) 906 and / or GPU(s) 908.

[0217] Examples of logic units 920 include one or more processing cores and / or components thereof, such as a data processing unit (DPU), a tensor core (TC), a tensor processing unit (TPU), a pixel vision core (PVC), a vision processing unit (VPU), a graphics processing cluster (GPC), a texture processing cluster (TPC), a streaming multi-processor (SM), a tree traversal unit (TTU), an artificial intelligence accelerator (AIA), a deep learning accelerator (DLA), an arithmetic logic unit (ALU), an application-specific integrated circuit (ASIC), a floating-point unit (FPU), an input / output (I / O) element, a peripheral component interconnect (PCI) or peripheral component interconnect express (PCIe) element, etc.

[0218] Communication interface 910 can include one or more receivers, transmitters, and / or transceivers that enable computing device 900 to communicate with other computing devices via electronic communication networks, including wired and / or wireless communications. Communication interface 910 can include components and functionality to enable communication over any of a plurality of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., through Ethernet or InfiniBand), low-power wide-area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, one or more logic units 920 and / or communication interface 910 can include one or more data processing units (DPUs) to transfer data received over a network and / or through interconnection system 902 directly to the memory of one or more GPU(s) 908.

[0219] I / O ports 912 can enable the computing device 900 to logically couple to other devices including I / O components 914, presentation components 918, and / or other components, some of which can be built in to (e.g., integrated in) the computing device 900. Illustrative I / O components 914 include a microphone, mouse, keyboard, joystick, game pad, game controller, satellite dish, scanner, printer, wireless device, etc. The I / O components 914 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by a user. In some instances, inputs can be transmitted to appropriate network elements for further processing. A NUI can implement any combination of speech recognition, gesture recognition, facial recognition, biometric recognition, posture recognition, gesture recognition within a

[0220] A power supply 916 can include a hard-wired power supply, a battery power supply, or a combination thereof. The power supply 916 can provide power to the computing device 900 to enable the components of the computing device 900 to operate.

[0221] The presentation components 918 can include a display (e.g., a monitor, a touch screen, a television, a heads-up display (HUD), other display types, or a combination thereof), speakers, and / or other presentation components. The presentation components 918 can receive data from other components (e.g., the GPU 908, the CPU 906, the DPU, etc.) and output the data (e.g., as a

[0222] Example data center

[0223] Figure 10 An example data center 1000 that can be used in at least one embodiment of the present disclosure is shown. The data center 1000 can include a data center infrastructure layer 1010, a framework layer 1020, a software layer 1030, and / or an application layer 1040.

[0224] As Figure 10As shown, the data center infrastructure layer 1010 can include a resource orchestrator 1012, grouped computing resources 1014, and node computing resources (“node C.R.s”) 1016(1)-1016(N), where “N” represents any whole, positive integer. In at least one embodiment, the node C.R.s 1016(1)-1016(N) can include, but are not limited to, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic random access memory), storage devices (e.g., solid state or disk drives), network input / output (NW I / O) devices, network switches, virtual machines (VMs), power modules, and / or cooling modules, etc. In some embodiments, one or more of the node C.R.s 1016(1)-1016(N) can correspond to a server having one or more of the above-described computing resources. Moreover, in some embodiments, the node C.R.s 1016(1)-1016(N) can include one or more virtual components, such as a vGPU, a vCPU, etc., and / or one or more of the node C.R.s 1016(1)-1016(N) can correspond to a virtual machine (VM).

[0225] In at least one embodiment, the grouped computing resources 1014 can include separate groupings of node C.R.s 1016 housed within one or more racks (not shown) or housed within many racks at different geographic locations (also not shown). The separate groupings of node C.R.s 1016 within the grouped computing resources 1014 can include grouped computing, network, memory, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, several node C.R.s 1016 including CPUs, GPUs, DPUs, and / or other processors can be grouped within one or more racks to provide computing resources to support one or more workloads. The one or more racks can also include any number of power modules, cooling modules, and / or network switches in any combination.

[0226] The resource orchestrator 1012 can configure or otherwise control the one or more node C.R.s 1016(1)-1016(N) and / or the grouped computing resources 1014. In at least one embodiment, the resource orchestrator 1012 can include a software design infrastructure (SDI) management entity for the data center 1000. The resource orchestrator 1012 can include hardware, software, or some combination thereof.

[0227] In at least one embodiment, as Figure 10As shown, the framework layer 1020 may include a job scheduler 1032, a configuration manager 1034, a resource manager 1036, and / or a distributed file system 1038. The framework layer 1020 may include a framework that supports the software 1032 of the software layer 1030 and / or one or more applications 1042 of the application layer 1040. The software 1032 or the application 1042 may include network-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. The framework layer 1020 may be, but is not limited to, a free and open source software network application framework (such as Apache Spark) that can utilize the distributed file system 1038 for large-scale data processing (e.g., "big data"). TM (hereinafter referred to as "Spark"). In at least one embodiment, the job scheduler 1032 may include a Spark driver to facilitate scheduling workloads supported by the different layers of the data center 1000. The configuration manager 1034 may be capable of configuring the different layers, such as the software layer 1030 and the framework layer 1020 (which includes Spark and a distributed file system 1038 for supporting large-scale data processing). The resource manager 1036 may be capable of managing clustered or grouped computing resources that are mapped to the distributed file system 1038 and the job scheduler 1032 or allocated to support the distributed file system 1038 and the job scheduler 1032. In at least one embodiment, the clustered or grouped computing resources may include the grouped computing resources 1014 at the data center infrastructure layer 1010. The resource manager 1036 may coordinate with the resource coordinator 1012 to manage these mapped or allocated computing resources.

[0228] In at least one embodiment, the software 1032 included in the software layer 1030 may include software used by at least a portion of the node CRs 1016(1)-1016(N), the grouped computing resources 1014, and / or the distributed file system 1038 of the framework layer 1020. The one or more types of software may include, but are not limited to, Internet web search software, email virus scanning software, database software, and streaming video content software.

[0229] In at least one embodiment, applications 1042 included in application layer 1040 can include one or more types of applications used by at least portions of node C.R.s 1016(1)-1016(N), grouped computing resources 1014, and / or distributed file systems 1038 of framework layer 1020. One or more types of applications can include, but are not limited to, any number of genomics applications, cognitive computing and machine learning applications including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments.

[0230] In at least one embodiment, any of configuration manager 1034, resource manager 1036, and resource orchestrator 1012 can implement any number and type of self-modification actions based on any quantity and type of data acquired in any technically feasible manner. Self-modification actions can free data center operators of data center 1000 from making potentially poor configuration decisions and can avoid underutilization and / or poor performance portions of data center.

[0231] According to one or more embodiments described herein, data center 1000 can include tools, services, software, or other resources to train one or more machine learning models or use one or more machine learning models to predict or infer information. For example, machine learning model(s) can be trained by computing weight parameters according to a neural network architecture using software and / or computing resources described above with respect to data center 1000. In at least one embodiment, trained or deployed machine learning models corresponding to one or more neural networks can be used to infer or predict information using resources described above with respect to data center 1000 by using weight parameters computed by one or more training techniques such as, but not limited to, those described herein.

[0232] In at least one embodiment, data center 1000 can use CPUs, application specific integrated circuits (ASICs), GPUs, FPGAs, and / or other hardware (or virtual computing resources corresponding thereto) to perform training and / or inference using resources described above. Further, one or more software and / or hardware resources described above can be configured as services that allow users to train or perform inference on information, such as image recognition, speech recognition, or other artificial intelligence services.

[0233] Example network environment

[0234] Network environments suitable for implementing embodiments of the present disclosure can include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. The client devices, servers, and / or other device types (e.g., each device) can be implemented on one or more instances of computing device(s) 900 - e.g., each device can include similar components, features, and / or functionality of computing device(s) 900. Further, where backend devices (e.g., servers, NAS, etc.) are implemented, the backend devices can be included as part of a data center 1000, an example of which is described in more detail herein with respect to FIG. 1, and can be implemented on one or more instances of computing device(s) 900. Figure 9 Figure 10 are implemented as part of a data center 1000, an example of which is described in more detail herein with respect to FIG. 1, and can be implemented on one or more instances of computing device(s) 900.

[0235] Components of the network environment can communicate with each other via a network, which can be wired, wireless, or both. The network can include multiple networks or one of multiple networks. For example, the network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the public switched telephone network (PSTN)), and / or one or more private networks. Where the network includes a wireless telecommunication network, components such as base stations, communication towers, or even access points (among other components) can provide wireless connectivity.

[0236] Compatible network environments can include one or more peer-to-peer network environments (in which case servers can not be included in the network environment) and one or more client-server network environments (in which case one or more servers can be included in the network environment). In a peer-to-peer network environment, functionality described herein for servers can be implemented on any number of client devices.

[0237] In at least one embodiment, the network environment can include one or more cloud-based network environments, distributed computing environments, combinations thereof, and the like. A cloud-based network environment can include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which can include one or more core network servers and / or edge servers. The framework layer can include a framework that supports one or more applications of a software layer and / or an application layer. The software or applications can include network-based service software or applications, respectively. In embodiments, one or more client devices can use the network-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer can be, without limitation, a free and open-source software web application framework as can be used for large-scale data processing (e.g., “big data”) using a distributed file system. ​

[0238] The cloud-based network environment can provide cloud computing and / or cloud storage that performs any combination of the computing and / or data storage functions described herein (or one or more portions thereof). Any of these different functions can be distributed across multiple locations from a central or core server (e.g., can be distributed across one or more data centers in a state, region, country, globally, etc.). The core server can designate at least a portion of the functions to an edge server if the connection to the user (e.g., client device) is relatively close to the edge server. The cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0239] The client device(s) can include at least some of the components, features, and functionality of the example computing device 900 described herein with respect to Figure 9 As examples and not by way of limitation, a client device can be implemented as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, video camera, surveillance device or system, vehicle, boat, spaceship, virtual machine, drone, robot, handheld communication device, hospital device, gaming device or system, entertainment system, vehicle computer system, embedded system controller, remote control, appliance, consumer electronic device, workstation, edge device, any combination of these depicted devices, or any other suitable device.

[0240] The present disclosure can be described in the general context of machine-usable instructions or computer code, including computer-executable instructions such as program modules, being executed by a computer or other machine, such as a personal data assistant or other handheld device. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. The present disclosure can be practiced in a variety of system configurations, including hand-held devices, consumer electronics, general- purpose computers, more specialty computing devices, etc. The present disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network.

[0241] As used herein, the term "and / or," with respect to a listing of two or more items, means that one, including only one of the items, or combinations of the items, can be employed. For example, "A and / or B" can include A alone, B alone, or A and B. Likewise, "at least one of A or B" can include A alone, B alone, or at least one of A and at least one of B. Further, "at least one of A and B" can include at least one of A, at least one of B, or at least one of A and at least one of B.

[0242] The subject matter of the present disclosure is described with specificity herein to meet statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the present disclosure has been presented for the purpose of illustration and description so as to enable others, skilled in the art, to employ the subject matter of the present disclosure in various ways. Furthermore, although the terms "step" and / or "block" can be used herein to connote different elements of methods employed, the terms should not be interpreted as implying any particular order among or between various steps herein disclosed unless and except the order of individual steps is explicitly described.

Claims

1. A processor comprising: one or more circuits to: compute, using a deep neural network (DNN) and based at least in part on first sensor data generated using one or more first sensors of a ego machine, a first feature map representing one or more first feature descriptors; associate, based at least in part on the one or more first feature descriptors, a first feature descriptor with a first object; compute, using the DNN and based at least in part on second sensor data generated using at least one of the one or more first sensors or one or more second sensors of the ego machine, a second feature map representing one or more second feature descriptors; associate, based at least in part on the one or more second feature descriptors, a second feature descriptor with a second object; determine, based at least in part on the first feature descriptor and the second feature descriptor, that the first object and the second object are a same object; and based at least in part on determining that the first object and the second object are the same object, perform, by the ego machine, one or more operations.

2. The processor of claim 1, wherein the first sensor data is generated using the one or more first sensors at a first timestamp and the second sensor data is generated using the one or more first sensors at a second timestamp.

3. The processor of claim 1, wherein the first sensor data is generated using the one or more first sensors concurrently with the second sensor data being generated using the one or more second sensors.

4. The processor of claim 1, wherein associating the first feature descriptor with the first object comprises: computing, based at least in part on the first sensor data, a bounding shape; determining a subset of the one or more first feature descriptors that is at least partially within the bounding shape; and generating the first feature descriptor using the subset. averaging values corresponding to the subset of the one or more first feature descriptors.

5. The processor of claim 4, wherein generating the first feature descriptor using the subset comprises:

6. The processor of claim 4, wherein computing the bounding shape is performed using one of the DNN, another DNN, or an object detection algorithm.

7. The processor of claim 1, wherein the one or more operations comprise tracking the same object.

8. The processor of claim 1, wherein each of the one or more first feature descriptors and each of the one or more second feature descriptors is represented using a one-dimensional feature vector.

9. The processor of claim 1, wherein the DNN is trained using a triplet loss function.

10. The processor of claim 1, wherein the DNN is further trained using at least one of batch-hard triplet mining or batch semi-hard triplet mining.

11. The processor of claim 1, wherein the processor is included in at least one of: a control system for an autonomous or semi-autonomous machine; ​ A perception system for an autonomous or semi-autonomous machine; A system for performing simulation operations; A system for performing deep learning operations; A system implemented using edge devices; A system implemented using robots; A system that merges one or more virtual machines (VMs); A system implemented at least in part in a data center; or A system implemented at least in part using cloud computing resources.

12. A system comprising: one or more processing units; and one or more memory units storing instructions that, when executed by the one or more processing units, cause the one or more processing units to perform operations comprising: computing one or more first feature vectors using a deep neural network (DNN) and based at least in part on first sensor data generated using one or more first sensors of a self machine; associating a first feature vector with a first object using the one or more first feature vectors; computing one or more second feature vectors using the DNN and based at least in part on second sensor data generated using the one or more first sensors or one or more second sensors of the self machine; associating a second feature vector with a second object based at least in part on the one or more second feature vectors; defining the first object and the second object as a same object based at least in part on the first feature vector and the second feature vector; and performing one or more operations by the self machine based at least in part on defining the first object and the second object as the same object.

13. The system of claim 12, wherein the first sensor data is generated at a first time using the one or more first sensors, and the second sensor data is generated at a second time using the one or more first sensors.

14. The system of claim 12, wherein the first sensor data is generated at a time using the one or more first sensors, and the second sensor data is generated at the time using the one or more second sensors.

15. The system of claim 12, wherein associating the first feature vector with the first object comprises: computing a boundary shape based at least in part on the first sensor data; determining a subset of the one or more first feature vectors that is at least partially within the boundary shape; and generating the first feature vector using the subset. averaging values corresponding to the subset of the one or more first feature vectors.

17. The system of claim 12, wherein:

16. The system of claim 15, wherein generating the first feature vector using the subset comprises: the operations further comprise computing a distance between the first feature vector and the second feature vector; and defining the first object and the second object as the same object based at least in part on the distance being less than a threshold distance.

18. The system of claim 17, wherein computing the distance is performed using at least one of a cosine distance, an LI distance, an L2 distance, or a K-Nearest Neighbor (KNN) distance. ​ ​ 19. The system of claim 12, wherein the system is included in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing deep learning operations; a system implemented using edge devices; a system implemented using robots; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

20. A method comprising: using training data, labeling a first bounding shape of a first frame and a second bounding shape of a second frame as corresponding to an object; using a deep neural network (DNN) to determine one or more first feature descriptors corresponding to first points within the first bounding shape of the first frame and one or more second feature descriptors corresponding to second points within the second bounding shape of the second frame; determining a first feature descriptor of the one or more first feature descriptors corresponding to an anchor point of the first points; using a loss function to compare the first feature descriptor to a second feature descriptor corresponding to a second point of the second points; and updating one or more parameters of the DNN based at least in part on the comparison.

21. The method of claim 20, wherein a loss computed using the loss function is based on a magnitude of a difference between a first value corresponding to the first feature descriptor and a second value corresponding to the second feature descriptor.

22. The method of claim 20, further comprising: using the training data, labeling a third bounding shape of a third frame as corresponding to another object; using the deep neural network (DNN) to determine one or more third feature descriptors corresponding to third points within the third bounding shape of the third frame; and using the loss function to compare the first feature descriptor to a third feature descriptor corresponding to one of the third points, wherein updating the one or more parameters of the DNN is further based at least in part on comparing the first feature descriptor to the third feature descriptor.

23. The method of claim 22, wherein the third frame corresponds to the second frame.

24. The method of claim 22, wherein a loss computed using the loss function is based on a magnitude of a difference between a first value corresponding to the first feature descriptor and a second value corresponding to the second feature descriptor.

25. The method of claim 20, wherein the loss function is a triplet loss function.

26. The method of claim 20, wherein the training data is selected using online triplet mining.

27. The method of claim 26, wherein the online triplet mining comprises at least one of batch hard online triplet mining or batch semi-hard online triplet mining.

28. A processor comprising: ​ ​ One or more circuits to track an object across two or more frames generated using sensor data and based at least on: determining that a first feature vector associated with a first object in a first sensor frame is within a threshold similarity of a second feature vector associated with a second object in a second sensor frame, wherein the first feature vector and the second feature vector are computed using a deep neural network (DNN); and defining, based at least on the determining, the first object and the second object as the object.

Images