Object tracking and time-to-collision estimation for autonomous systems and applications

Abstract

The present disclosure relates to object tracking and time-to-collision estimation for autonomous systems and applications. In various examples, systems and methods for tracking an object and determining a time-to-collision value associated with the object are described. For example, the systems and methods can use a feature point associated with an object depicted in a first image and a feature point associated with a second image to determine a scalar change associated with the object. The systems and methods can then use the scalar change to determine a translation associated with the object. Using the scalar change and the translation, the systems and methods can determine that the object is also depicted in the second image. The systems and methods can further use the scalar change and a time baseline to determine a time-to-collision associated with the object. Upon performing the determination, the systems and methods can output data representative of at least an identifier of the object, a location of the object, and / or the time-to-collision.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 339,371, filed May 6, 2022, which is hereby incorporated by reference in its entirety. BACKGROUND

[0003] Vehicles, such as semi-autonomous vehicles and autonomous vehicles, use perception systems to track objects or actors in the surrounding environment. Conventional systems can use different techniques to track objects, such as using a Bayesian filter (e.g., Kalman filter (KF), particle filter, etc.) or a deep neural network (DNN) (e.g., convolutional neural network). However, both of these techniques heavily rely on object detectors to perform object tracking, which can result in less accurate or less reliable results. For example, some Bayesian filter techniques define the motion of an object from one image to the next, such as by using a state transition vector of the tracked object between images. Thus, when an object is first observed in an image, the state transition vector is initialized to predetermined values, and these values are revised based on subsequent detections corresponding to one or more subsequent images. The initialization of the predetermined values is a challenging task, at least due to the difficulty of predicting object velocities, and because of the reliance on these initialized values in subsequent determinations, this challenge can result in lower than desired or required accuracy or precision of object trajectories.

[0004] Furthermore, some DNN techniques attempt to use the feature encoding capabilities of a DNN for object tracking. For example, most existing DNN models initially train DNN features in an object detection pipeline. Then, existing DNN models use off-the-shelf matching / association algorithms to process the DNN features in order to match the DNN features to different objects. However, in some cases, both the learned DNN features and the association algorithms are directly optimized for the object tracking task, which can result in identity switching between objects - e.g., due to an object being occluded in a frame or otherwise losing track of an object in one or more frames. Furthermore, extracting and caching DNN features for each detected object can be computationally expensive. As such, these DNN models can not be able to scale to situations where there are a large number of objects to track in the environment.

[0005] Vehicles also use these perception systems to determine time-to-collision (TTC) estimates associated with tracked objects within the environment. For example, conventional systems can initially track an object, such as by using one or more of the processes described above, and then independently determine a TTC estimate associated with the object. To determine the TTC estimate, the conventional system can use the tracked object position as input to a model that determines a three-dimensional (3D) velocity and a 3D position of the tracked object within the environment. Using the 3D velocity and the 3D position of the object, the conventional system then post-processes to compute a TTC estimate value associated with the tracked object. SUMMARY

[0006] Embodiments of the present disclosure relate to techniques for tracking an object and determining a time-to-collision (TTC) estimate associated with the object using feature points. For example, the disclosed systems and methods determine a set of feature points of an object depicted in a plurality of (e.g., consecutive) images. The systems and methods then use the sets of feature points to track the object from one image to the next. For example, the systems and methods can use the sets of feature points to determine at least a translation and a scalar change associated with the object. The systems and methods can then use the translation, the scalar change, and a bounding shape associated with the object depicted in the images to predict a bounding shape associated with another subsequent image. Further, the systems and methods can use the predicted bounding shape to determine that an object detected in the other image is associated with the tracked object.

[0007] The systems and methods described herein can further use the feature points to determine a TTC estimate associated with the object. For example, the feature points can be used to determine respective scalar changes associated with different pairs of feature points. The scalar changes can then be used to determine a final scalar change associated with the object over a (e.g., time baseline). To determine the TTC estimate, one or more algorithms can be used to determine the TTC estimate based on the final scalar change over the time baseline.

[0008] In some embodiments, the current system is able to track the position of an object using a set of feature points associated with an image as compared to conventional systems, such as the system described above. As such, the current system does not require or set any initial state transition vector or Bayesian filter to track the state of an object and the motion of the object - which results in an improvement in tracking precision and accuracy as compared to conventional systems. Furthermore, the system and method of the present disclosure can determine a TTC estimate associated with an object using a set of feature points associated with an image as compared to conventional systems. As such, the current system can not use any two-dimensional (2D) or 3D projections to determine a TTC estimate, but rather rely on data corresponding to image space to determine a TTC estimate - which can further improve the accuracy and precision of the current system as compared to conventional systems, while also reducing computation and latency. BRIEF DESCRIPTION OF DRAWINGS

[0009] The present system and method for tracking an object and determining a time-to-collision estimate associated with the object is described in detail below with reference to the accompanying drawings, in which:

[0010] Figure 1 An example data flow diagram of a process for tracking an object and determining a time-to-collision (TTC) estimate associated with the object according to some embodiments of the present disclosure is shown;

[0011] Figure 2 An example of feature points associated with an object depicted in an image according to some embodiments of the present disclosure is shown;

[0012] Figure 3 An example of determining an enclosing shape associated with an object according to some embodiments of the present disclosure is shown;

[0013] Figures 4A-4D An example of determining a predicted state of an object using feature points according to some embodiments of the present disclosure is shown;

[0014] Figures 5A-5B An example of associating a detected object with a tracked object according to some embodiments of the present disclosure is shown;

[0015] Figure 6 An example of using a confidence score to determine various states of a tracked object according to some embodiments of the present disclosure is shown;

[0016] Figure 7 An example of determining a time-to-collision estimate associated with an object according to some embodiments of the present disclosure is shown;

[0017] Figure 8 An example of verifying a scalar change used to determine a TTC estimate according to some embodiments of the present disclosure is shown;

[0018] Figure 9 is a flowchart showing a method for tracking an object using feature points according to some embodiments of the present disclosure;

[0019] Figure 10 is a flowchart showing a method for determining a TTC estimate associated with an object according to some embodiments of the present disclosure;

[0020] Figure 11A is an illustration of an example autonomous vehicle according to some embodiments of the present disclosure;

[0021] Figure 11B is an example of a camera position and field of view of an example autonomous vehicle according to some embodiments of the present disclosure; Figure 11A

[0022] Figure 11C is an example of a camera position and field of view of an example autonomous vehicle according to some embodiments of the present disclosure; Figure 11A is a block diagram of an example system architecture of an example autonomous vehicle according to some embodiments of the present disclosure;

[0023] Figure 11D is a system diagram for communicating between a cloud-based server and an example autonomous vehicle according to some embodiments of the present disclosure; Figure 11A

[0024] Figure 12 is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure; and

[0025] Figure 13 is a block diagram of an example data center suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0026] Systems and methods related to tracking objects and determining a time-to-collision (TTC) estimate associated with an object using feature points are disclosed. Although can be directed to an example autonomous or semi-autonomous vehicle 1100 (alternatively referred to herein as "vehicle 1100" or "ego-machine 1100," examples of which are described with respect to Figures 11A-11D ​​The present disclosure is described, which is not meant to be limiting. For example, the systems and methods described herein can be used by, but are not limited to, non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more adaptive driving assistance systems (ADAS)), driving and non-driving robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles connected to one or more trailers, dirigibles, ships, shuttles, emergency vehicles, motorcycles, electric or motorized bicycles, airplanes, construction vehicles, underwater vehicles, drones, and / or other vehicle types. Moreover, although the present disclosure can be described with respect to object tracking and / or TTC estimation, this is not meant to be limiting, and the systems and methods described herein can be used for augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications, and / or any other technical space that can use object tracking or TTC (or other impact or collision determination).

[0027] For example, the systems and methods described herein can determine respective sets of feature points associated with an object depicted in a plurality of (e.g., consecutive, time-intervalled, etc.) images. To determine the sets of feature points, a first enclosing shape associated with the object can initially be determined for a first image. The first enclosing shape can include feature points associated with the first image, where at least a portion of the feature points are associated with the object and a portion of the feature points can be associated with a background surrounding the object. A tracking shape associated with the object can be determined, such as by reducing a size of the previously determined enclosing shape. Using the tracking shape, a set of feature points associated with a second, subsequent image can be identified. Moreover, the set of feature points can be used to identify a corresponding set of feature points from the first image. In this way, the systems and methods described herein can identify sets of feature points that include feature points associated with the object, without including feature points associated with a background or environment surrounding the object.

[0028] The systems and methods can then use the sets of feature points to determine a state change associated with the object from the first image (e.g., at time = t - b, where b is a time baseline) to the second image (e.g., at time = t). For example, the sets of feature points can be used to determine one or more parameters associated with the object, such as a translation and a scalar change. To determine these parameters, a first distance between a pair of feature points in the first image and a second distance between a corresponding pair of feature points in the second image are determined. A scalar change between the pair of feature points is then determined, such as by subtracting the second distance from the first distance, and a final scalar change is determined using the scalar change. In some examples, the final scalar change can be determined based on a mean of the scalar changes, a median of the scalar changes, a mode of the scalar changes, and / or using one or more additional and / or alternative techniques.

[0029] The final scalar change can then be used to determine a translation of one or more of the feature points. For example, a translation of a feature point can be determined by multiplying the feature point in the first image by the final scalar change and then subtracting the feature point location in the second image. This process can be applied to one or more of the feature points in the set of feature points in order to determine a translation of that feature point. Further, the translation of that feature point can be used to determine a final translation. In some examples, the final translation can be determined based on an average of the translations, a median of the translations, a mode of the translations, and / or using one or more additional and / or alternative techniques.

[0030] The system and method can then track the object from the first image to the second image using the first enclosing shape associated with the first image, the final scalar change, and the final translation. For example, a predicted enclosing shape of the object can be determined using the first enclosing shape associated with the first image, the final scalar change, and the final translation. In some examples, the predicted enclosing shape can be determined by multiplying values corresponding to the first enclosing shape (e.g., size, dimensions, coordinates of points, etc.) by the final scalar change and then moving the enclosing shape based on the translation. The predicted enclosing shape and a second enclosing shape associated with the detected object can then be used to determine that the object detected using the second image is associated with the object from the first image (e.g., the tracked object). These processes can then be repeated in any number of iterations in order to track the object in multiple images— such as until the object is no longer detected in a certain threshold number of subsequent frames (e.g., one frame, ten frames, fifteen frames, etc.).

[0031] As described herein, in some examples, the system and method can further determine a TTC estimate associated with the object using the set of feature points. For example, the system and method can perform one or more of the processes described above to determine scalar changes for pairs of feature points from the first image to the second image. In some examples, one or more of the scalar changes associated with one or more distances that do not satisfy (e.g., are less than) a threshold distance can be filtered out. The final scalar change for the TTC estimate can then be determined using the remaining scalar changes. In some examples, the final scalar change is determined based on an average of the scalar changes, a median of the scalar changes, a mode of the scalar changes, and / or using one or more additional and / or alternative techniques. The system and method can then determine the TCC estimate based on the final scalar change and a time baseline between the first image and the second image using one or more algorithms. In some examples, and as described in greater detail herein, one or more smoothing operations can be used to determine the final TTC.

[0032] The systems and methods described herein can be used by, but are not limited to, non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more adaptive driving assistance systems (ADAS)), driving and non-driving robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles connected with one or more trailers, dirigibles, ships, shuttles, emergency vehicles, motorcycles, electric or motorized bicycles, airplanes, construction vehicles, underwater vehicles, drones, and / or other vehicle types. Further, the systems and methods described herein can be used for various purposes, by way of example and without limitation, machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications, deep learning, environment simulation, data center processing, conversational AI, light transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation for 3D assets, cloud computing, and / or any other suitable application.

[0033] The disclosed embodiments can be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines), systems implemented using robots, aviation systems, medical systems, boating systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems implemented using edge devices, systems incorporating one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in data centers, systems for performing conversational AI, systems for performing light transport simulation, systems for performing 3D asset collaborative content creation, systems implemented at least partially using cloud computing resources, and / or other types of systems.

[0034] Figure 1 An example data flow diagram illustrating a process 100 for tracking an object and determining a time-to-collision (TTC) estimate associated with the object is shown in accordance with some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are set forth only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, commands, groupings of functions, etc.) can be used in addition to or instead of those shown, and some elements can be wholly omitted altogether. Further, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in conjunction with other components, and in any suitable combination and location. The various functions described herein as being performed by the entities can be stored in and executed by hardware, firmware and / or software. For example, the various functions can be performed by a processor executing instructions stored in memory. In some embodiments, the systems, methods, and processes described herein can use components and / or functionality similar to those of Figures 11A-11D an example autonomous vehicle 1100, Figure 12 an example computing device 1200, and / orFigure 13 similar components, features, and / or functions of the example data center 1300 to perform.

[0035] The process 100 can include a tracking component 102 that tracks objects within an environment. For example, the tracking component 102 can include a prediction component 104 that processes detected object data 106 and feature point data 108 to predict new states of objects on an image (or other sensor data representation, such as a point cloud, projected image, etc.) represented by sensor data (e.g., image data). In some examples, a perception system of a vehicle initially detects objects within an image and then sends detected object data 106 and feature point data 108 to the tracking component 102.

[0036] For example, Figure 2 An example of feature points 202(1)-(24) (also referred to singularly as “feature point 202” or plural as “feature points 202”) associated with an image 204 represented by sensor data 206 (which can be represented by feature point data 108) is shown in accordance with some embodiments of the present disclosure. In some examples, a perception system 208 of a vehicle initially processes sensor data 108, which can include image data. The image data can include data representing an image that depicts one or more fields of view of one or more cameras (e.g., image sensors) of a vehicle, such as stereo cameras, wide-angle cameras (e.g., fisheye cameras), infrared cameras, surround cameras (e.g., 360-degree cameras), long-range and / or mid-range cameras, and / or other camera types of an autonomous vehicle. In some examples, the image data can be captured in one format (e.g., RCCB, RCCC, RBGC, etc.) and then converted to another format (e.g., during pre-processing of the image data). In other examples, the image data can be provided as input to a sensor data pre-processor (not shown) to generate pre-processed image data (discussed herein). Many types or formats of images can be used as input; for example, compressed images such as Joint Photographic Experts Group (JPEG), Red Green Blue (RGB), or Luma / Chroma (YUV) formats, compressed images as frames sourced from a compressed video format such as H.264 / Advanced Video Coding (AVC) or H.265 / High Efficiency Video Coding (HEVC), raw images such as sourced from Red Clear Blue (RCCB), Red Clear (RCCC), or other types of imaging sensors. In some examples, different formats and / or resolutions can be used for training a neural network, rather than for inference (e.g., during deployment of a machine learning model in an autonomous vehicle).

[0037] The perception system 208 can use one or more techniques to process the sensor data 206 in order to determine the feature points 202. These techniques can include, but are not limited to, Harris Corners, Scale Invariant Feature Transform (SIFT), Speeded Up Robust Feature (SURF), Features from Accelerated Segment Test (FAST), Oriented FAST and Rotated BRIEF (ORB), and / or any other technique. While Figure 2 The example of FIG. 2A shows twenty-four feature points 202 associated with the image 204, but in other examples, an image can be associated with any number of feature points (e.g., one feature point, five feature points, fifty feature points, one hundred feature points, one thousand feature points, etc.). Moreover, while the example of FIG. 2A shows the perception system 208 separate from the tracking component 102 and the prediction component 104, in other examples, the perception system 208 can be included as part of the tracking system 102 and / or the prediction component. Figures 1-2 The example of FIG. 2A shows twenty-four feature points 202 associated with the image 204, but in other examples, an image can be associated with any number of feature points (e.g., one feature point, five feature points, fifty feature points, one hundred feature points, one thousand feature points, etc.). Moreover, while the example of FIG. 2A shows the perception system 208 separate from the tracking component 102 and the prediction component 104, in other examples, the perception system 208 can be included as part of the tracking system 102 and / or the prediction component.

[0038] The perception system 208 can also include functionality for performing object detection, segmentation, and / or classification. For example, the perception system 208 can output object data 106 indicative of detected lanes and boundaries on a driving surface, detected drivable free space, detected poles or signs, detected traffic lights, detected objects in the environment (e.g., vehicles, pedestrians, animals, inanimate objects, etc.), detected waiting conditions and intersections, and / or the like. In additional or alternative examples, the perception system 208 can generate object data 106 indicative of one or more characteristics associated with a detected object and / or the environment in which the object is located. Characteristics associated with an object can include, but are not limited to, x-position (global and / or local position), y-position (global and / or local position), z-position (global and / or local position), orientation (e.g., roll, pitch, yaw), object classification (e.g., type of object), velocity, acceleration, extent (size) of the object, etc.

[0039] In those embodiments in which the perception system 208 performs detection, the perception system 208 can generate output data 106 indicative of detections of objects detected in the image. Such detections can include two-dimensional bounding shapes and / or masks of the detected objects. Further, in some examples, the output data 106 can be indicative of probabilities associated with the objects, such as probabilities associated with locations of the objects, classifications of the objects, and / or the like. In some examples, the detections of the perception system 208 can use a machine learning method (e.g., scale-invariant feature transform (SIFT), histogram of oriented gradient (HOG), etc.) followed by a support vector machine (SVM) to classify objects depicted in the image represented by the sensor data 206. Further, or alternatively, the detections can utilize a deep learning method based on a convolutional neural network (CNN) to classify objects depicted in the image represented by the sensor data 206.

[0040] For example, Figure 3 An example is shown in which the perception system 208 determines a bounding shape 302 associated with an object 304 depicted in the image 204, in accordance with some embodiments of the present disclosure. In some examples, the perception component 206 can determine the bounding shape 302 using one or more object recognition and / or computer vision techniques. The object recognition and / or computer vision techniques can include, but are not limited to, SURF, FAST, Faster R-CNN, You Only Look Once (YOLO), histogram of oriented gradient (HOG), Spatial Pyramid Pooling (SPP-net), and / or any other technique. While the example shows the bounding shape 302 including a rectangle, in other examples, the bounding shape 302 can include any other shape (e.g., a circle, a triangle, a square, a hexagon, etc.). Figure 3 An example is shown in which the perception system 208 determines a bounding shape 302 associated with an object 304 depicted in the image 204, in accordance with some embodiments of the present disclosure. In some examples, the perception component 206 can determine the bounding shape 302 using one or more object recognition and / or computer vision techniques. The object recognition and / or computer vision techniques can include, but are not limited to, SURF, FAST, Faster R-CNN, You Only Look Once (YOLO), histogram of oriented gradient (HOG), Spatial Pyramid Pooling (SPP-net), and / or any other technique. While the example shows the bounding shape 302 including a rectangle, in other examples, the bounding shape 302 can include any other shape (e.g., a circle, a triangle, a square, a hexagon, etc.).

[0041] The prediction component 104 can then use the feature point data 108 representing the feature points 202 and the object data 106 representing at least the bounding shape 302 to predict a new state of the object 304. For example, Figures 4A-4D An example is shown in which the perception system 208 determines a bounding shape 302 associated with an object 304 depicted in the image 204, in accordance with some embodiments of the present disclosure. In some examples, the perception component 206 can determine the bounding shape 302 using one or more object recognition and / or computer vision techniques. The object recognition and / or computer vision techniques can include, but are not limited to, SURF, FAST, Faster R-CNN, You Only Look Once (YOLO), histogram of oriented gradient (HOG), Spatial Pyramid Pooling (SPP-net), and / or any other technique. While the example shows the bounding shape 302 including a rectangle, in other examples, the bounding shape 302 can include any other shape (e.g., a circle, a triangle, a square, a hexagon, etc.). Figure 4AAs illustrated in the examples, prediction component 104 can use the bounding shape 302 to determine the tracked shape 402 within a subsequent image 402. In some examples, prediction component 104 can determine the tracked shape 402 by reducing the size of the bounding shape 302 (e.g., by a scalar value). In some examples, prediction component 104 can use one or more factors to determine the tracked shape 304 (e.g., scalar values). As described herein, factors may include, but are not limited to, object classification, distance to the object, object size (e.g., as depicted in image 204), and / or any other factors.

[0042] As shown, the tracked shape 402 includes feature points 406(1)-(6) (also referred to as "feature points 406" in the singular or plural) determined for image 404, such as by using a process similar to that for feature points 202 (e.g., using a perception system 208). The prediction component 104 can then use the feature points 406 from image 204 and the corresponding feature points 202 to predict the new state of object 304. In some examples, the prediction component 104 can use at least a threshold number of feature points 406 to predict the new state of object 304. The threshold number of feature points 406 can include, but is not limited to, two feature points 406, five feature points 406, ten feature points 406, fifty feature points 406, and / or any other number of feature points 406.

[0043] exist Figure 4A In the example, prediction component 104 (and / or perception system 208) may use one or more of the processes described herein to determine feature points 202(1)-(6) corresponding to feature point 406. For example, as shown, prediction component 104 may determine that feature point 406(1) corresponds to feature point 202(1), feature point 406(2) corresponds to feature point 202(2), feature point 406(3) corresponds to feature point 202(3), feature point 406(4) corresponds to feature point 202(4), feature point 406(5) corresponds to feature point 202(5), and feature point 406(6) corresponds to feature point 202(6), which in Figure 4A In the example, the arrows indicate this. In other words, the prediction component 104 can determine that feature points 406(1)-(6) from image 404 are the same feature points as feature points 202(1)-(6) from image 204.

[0044] like Figure 4AAs further shown by the example of FIG. 4, by using the tracked shape 402 to determine the feature points 406, the prediction component 104 can not consider feature points 202 associated with other objects and / or the background when predicting the new state of the object 304. For example, the prediction component 104 can not consider the feature point 202(11) (and / or similar feature points 202(12)-(24), not shown for clarity reasons) when predicting the new state of the object 304. Further, the prediction component 104 can not consider at least some feature points 202 associated with the object 304. For example, the prediction component 104 can not consider the feature point 202(7) (and / or similar feature points 202(8)-(9), not shown for clarity) when predicting the new state of the object 304. However, the prediction component 104 can still be able to predict the new state of the object 304 with the remaining feature points 202(1)-(6) associated with the object 304.

[0045] The prediction component 104 can then determine scalar changes for pairs of feature points 202, 406. For example, and as shown by the example of FIG. 4, the prediction component 104 can determine a first distance 408(1) between the feature point 202(1) and the feature point 202(2) and a second distance 408(2) between the feature point 406(1) and the feature point 406(2) for the pair of feature points 202(1)-(2), 406(1)-(2). The prediction component 104 can then determine a scalar change associated with the pair of feature points 202(1)-(2), 406(1)-(2) based on the first distance 408(1) and the second distance 408(2). For example, in some examples, the prediction component 104 can determine the scalar change associated with the pair of feature points 202(1)-(2), 406(1)-(2) by taking the difference between the second distance 408(2) and the first distance 408(1). The prediction component 104 can then perform a similar process to determine a scalar change associated with another pair of feature points 202, 406. For example, and in some examples, the prediction component 104 can determine a scalar change for each pair of feature points 202, 406. Figure 4B As shown by the example of FIG. 4, the prediction component 104 can generate a list 410 of scalar changes 412(1)-(15) (also referred to singularly as “scalar change 412” or in the plural as “scalar changes 412”) associated with pairs 414(1)-(15) (also referred to singularly as “pair 414” or in the plural as “pairs 414”) of feature points 202, 406. In some examples, the prediction component 104 can generate the list 410 of scalar changes 412(1)-(15) associated with the pairs 414(1)-(15) of feature points 202, 406 by determining a scalar change for each pair 414(1)-(15) of feature points 202, 406.

[0046] Figure 4C As shown by the example of FIG. 4, the prediction component 104 can generate a list 410 of scalar changes 412(1)-(15) (also referred to singularly as “scalar change 412” or in the plural as “scalar changes 412”) associated with pairs 414(1)-(15) (also referred to singularly as “pair 414” or in the plural as “pairs 414”) of feature points 202, 406. In some examples, the prediction component 104 can generate the list 410 of scalar changes 412(1)-(15) associated with the pairs 414(1)-(15) of feature points 202, 406 by determining a scalar change for each pair 414(1)-(15) of feature points 202, 406. Figure 4C ​In the example, prediction component 104 identifies 15 scalar variations 412 because there are 15 distinct combinations of pairs 414 of feature points 202, 406. However, in other examples, as described herein, prediction component 104 can identify scalar variations 412 for all combinations of pairs 414 of feature points 202, 406. Prediction component 104 can then use list 410 to determine the final scalar variation associated with object 304.

[0047] In some examples, before and / or simultaneously with determining the final scalar change, the prediction component 104 may initially filter out pairs 414 of feature points 202, 406 that are associated with distances that do not meet (e.g., are less than) a threshold distance. For example, and referring to... Figure 4B The prediction component 104 can determine a third distance 408(3) between feature points 202(5) and 202(6) and / or a fourth distance 408(4) between feature points 406(5) and 406(6). Then, the prediction component 104 can determine that the third distance 408(3) and / or the fourth distance 408(4) are less than a threshold distance. Therefore, the prediction component 104 may not initially determine the scalar change 412(15) associated with the pair 414(15) including feature points 202(5)-(6) and 406(5)-(6) and / or may filter out the scalar change 412(15) when determining the final scalar change.

[0048] The prediction component 104 can then use the scalar change 412 (and / or the filtered residual scalar change 412) to determine the final scalar change. As described herein, the prediction component 104 may determine the final scalar change as the mean, median, mode, and / or one or more additional and / or alternative techniques of the scalar change 412. The prediction component 104 can then use the final scalar change to determine the translation associated with object 304 between image 204 and image 404.

[0049] For example, the prediction component 104 can determine one or more translations associated with one or more of the feature points 202(1)-(6). To determine the translation associated with a feature point 202, the prediction component 104 can multiply the feature point 202 by the final scalar change and then subtract that result from the location of the corresponding feature point 406. For example, and with respect to feature point 202(1), the prediction component 104 can determine the translation by multiplying the feature point 202(1) by the final scalar change and then subtracting that result from the location of the feature point 406(1) corresponding to the feature point 202(1). In some examples, the prediction component 104 can perform a similar process to determine the translation of one or more (e.g., all) of the other feature points 202(2)-(6). The prediction component 104 can then use the translations to determine a final translation associated with the object 304. As described herein, the prediction component 104 can determine the final translation as the mean of the translations, the median of the translations, the mode of the translations, and / or using one or more additional and / or alternative techniques.

[0050] Next, and as shown in the example of FIG. 4B, the prediction component 104 can then use the enclosing shape 302, the final scalar change, and the final translation to determine a new enclosing shape 416 (not shown in the example of FIG. 4B for clarity reasons) of the object 304 depicted in the image 404. In some examples, the prediction component 104 determines the predicted enclosing shape 416 by multiplying the enclosing shape 302 by the final scalar change and then adding the final translation. Thus, the prediction component 104 can determine the size of the predicted enclosing shape 416 as the size of the enclosing shape 302 multiplied by the final scalar change. The prediction component 104 can then determine the location of the predicted enclosing shape 416 by moving the predicted enclosing shape 416 in the x- and y-directions associated with the final translation. In the example of FIG. 4B, the predicted enclosing shape 416 can represent a predicted state of the object 304. Figure 4D Figure 4D Figures 4A-4D

[0051] Although referred to herein as a scalar change, in embodiments, the change can be represented using something other than a scalar value. For example, the change can be reflected using a vector, a tensor, and / or other representation methods.

[0052] With reference to FIG. 4A, Figure 1 ​​​Process 100 may include an association component 110 of prediction component 102 determining an association between a tracked object and objects detected within (e.g., subsequently) the image. In some examples, association component 110 uses the determined location of the detected object (e.g., using object data 106) and the predicted location of the tracked object (e.g., the predicted bounding shape 416) to determine a score for the detected object. In some examples, association component 110 determines more than one score for the detected object, where each score associates the detected object with a corresponding predicted location of the tracked object. For example, if five objects are detected within the image, and tracking component 102 is currently tracking four objects, then association component 110 may determine twenty scores.

[0053] For example, Figure 5A Examples of determining scores for detected objects according to some embodiments of this disclosure are shown, wherein the scores are subsequently used to associate the detected objects with tracked objects. Figure 5A As illustrated in the examples, it is assumed that component 110 can use the predicted bounding shape 416 determined by prediction component 104 and the determined bounding shape 502(1) for the detected object 504(1) to determine a first score that the detected object 504(1) is the tracked object 304. In some examples, the bounding shape 502(1) is determined using one or more processes similar to the bounding shape 302 (e.g., using perception system 208). In some examples, association component 110 can use the intersection over union (IoU) to determine the first score. For example, the greater the overlap between the bounding shape 502(1) and the predicted bounding shape 416, the higher the first score. Furthermore, the smaller the overlap between the bounding shape 502(1) and the predicted bounding shape 416, the lower the first score. However, in other examples, association component 110 can use one or more additional and / or alternative techniques to determine the first score.

[0054] exist Figure 5AIn the example of FIG. 5, the association component 110 can further determine a second score that the detected object 504(1) is associated with another tracked object using the predicted bounding shape 506 of the other tracked object and the bounding shape 502(1) of the detected object 504(1). In addition, the association component 110 can determine a third score that the detected object 504(2) is a tracked object associated with the predicted bounding shape 416 using the bounding shape 502(2) associated with the detected object 504(2) and the predicted bounding shape 416. Finally, the association component 110 can determine a fourth score that the detected object 504(2) is a tracked object associated with the predicted bounding shape 504 using the bounding shape 502(2) associated with the detected object 504(2) and the predicted bounding shape 504. Thus, since there are two detected objects 504(1)-(2) and two tracked objects associated with the predicted bounding shapes 416, 504, the association component 110 can determine four scores.

[0055] In some examples, and as shown in the example of FIG. 5, the association component 110 can generate an association table 510 including scores 512(1)-(4) (also referred to in the singular as “score 512” or in the plural as “scores 512”) associated with the detected objects 504(1)-(2). For example, and as shown in the example of FIG. 5, the association table 510 can include a first score 512(1) that the detected object 504(1) is associated with the tracked object (e.g., object 304) associated with the predicted bounding shape 416, a second score 512(2) that the detected object 504(1) is associated with the tracked object associated with the predicted bounding shape 504, a third score 512(3) that the detected object 504(2) is associated with the tracked object associated with the predicted bounding shape 416, and a fourth score 512(4) that the detected object 504(2) is associated with the tracked object associated with the predicted bounding shape 504. Figure 5B Figure 5B In some examples, and as shown in the example of FIG. 5, the association component 110 can generate an association table 510 including scores 512(1)-(4) (also referred to in the singular as “score 512” or in the plural as “scores 512”) associated with the detected objects 504(1)-(2). For example, and as shown in the example of FIG. 5, the association table 510 can include a first score 512(1) that the detected object 504(1) is associated with the tracked object (e.g., object 304) associated with the predicted bounding shape 416, a second score 512(2) that the detected object 504(1) is associated with the tracked object associated with the predicted bounding shape 504, a third score 512(3) that the detected object 504(2) is associated with the tracked object associated with the predicted bounding shape 416, and a fourth score 512(4) that the detected object 504(2) is associated with the tracked object associated with the predicted bounding shape 504.

[0056] ​For example, in some examples, the association component 110 can determine that the detected objects 504(1)-(2) are tracked objects associated with the highest scores 512. For example, the association component 110 can determine that the detected object 504(1) is a tracked object associated with the predicted bounding shape 416 that can include the object 304 based on the first score 512(1) being greater than the second score 512(2). In some examples, the association component 110 can set a minimum threshold score for associating detected objects 504(1)-(2) with tracked objects. For example, and using the example of Figures 5A-5B the association component 110 can determine that the third score 512(3) and the fourth score 512(4) are less than the minimum threshold score. Accordingly, the association component 110 can determine that the detected object 504(2) is not associated with a tracked object (e.g., the detected object 504(2) includes a new object, which is described in detail herein).

[0057] Referring to Figure 1 , the process 100 can include an update component 112 of the tracking component 102 updating a tracking shape of a tracked object if an association of the tracked object is identified. For example, and using the example of Figures 2-5B the tracking shape 416 associated with the object 304 can be updated using the bounding shape 502 based on the detected object 504(1) within the image 404 being associated with the object 304. In this way, the tracking component 102 is able to continue tracking the location of the object 304 as sensor data 106 representing new images of the object 304 are received by the tracking component 102.

[0058] Additionally, in some examples, the update component 112 is configured to update a respective confidence score of one or more (e.g., each) of the tracked objects. For example, the update component 112 can determine an initial confidence score for a new object when the object (e.g., the detected object 504(2)) is initially identified by the tracking component 102 using the processes described herein. The update component 112 can then increase or decrease the confidence score of a tracked object based on whether an association is formed with a detected object in an image. For example, the update component 112 can increase the confidence score of a tracked object (in embodiments, not exceeding a maximum confidence value) each time the association component 110 is able to associate the tracked object with a detected object in an image. For example, and using the example of Figures 2-5BFor example, update component 112 may update the confidence score associated with object 304 based on associating object 304 depicted in image 204 with detected object 504(1) from image 404. Furthermore, update component 112 may reduce the confidence score (in embodiments, not below the minimum confidence value) each time association component 110 fails to associate the tracked object with the detected object from the image.

[0059] In some examples, the update component 112 may update the confidence score associated with the tracked object linearly. For example, when the association component 110 associates a detected object with the tracked object, the association component 112 may increase the confidence score associated with the tracked object by, for example, a set amount. Furthermore, when the association component 110 cannot associate a detected object with the tracked object, the association component 112 may decrease the confidence score associated with the tracked object by, for example, a set amount (the same or different amount used for positive association). In some examples, the update component 112 may update the confidence score associated with the tracked object non-linearly. In such examples, the update component 112 may use one or more factors when updating the confidence score. These one or more factors may include, but are not limited to, whether the object is occluded within the image, the number of feature points associated with the tracked object, the number of feature points associated with the detected object, the classification of the tracked object, the distance to the tracked object, and / or similar factors.

[0060] In some examples, the update component 112 can be further configured to use confidence scores to update one or more states associated with the tracked object. For example, Figure 6 Examples are shown of updating the state associated with the tracked object using confidence score 602 within time period 604, according to some examples of this disclosure. For example... Figure 6 As shown in the example, the confidence score 602 is initially zero at time T(0) because the tracking component 102 has not yet detected any tracked object associated with the confidence score 602. The tracking component 102 can then continue using the image between time T(0) and time T(2) to detect tracked objects. Therefore, and as shown, the updating component 112 can begin increasing the confidence score 602 associated with the tracked object between time T(0) and time T(2).

[0061] Next, the tracking component 102 can no longer use images between time T(2) and time T(3) to detect the tracked object. Accordingly, as illustrated, the updating component 112 can begin to decrease the confidence score 602 associated with the tracked object between time T(2) and time T(3). Next, the tracking component 102 can again use images between time T(3) and time T(4) to detect the tracked object. Accordingly, as illustrated, the updating component 112 can again increase the confidence score 602 associated with the tracked object between time T(3) and time T(4). Finally, the tracking component 102 can again no longer use images between time T(4) and time T(7) to detect the tracked object. Accordingly, and as illustrated, the updating component 112 can begin to decrease the confidence score 602 associated with the tracked object between time T(4) and time T(7).

[0062] As Figure 6 As further illustrated by the example of FIG. 6, the updating component 112 can use thresholds 606(1)-(3) (also referred to in the singular as “threshold 606” or in the plural as “thresholds 606”) to determine a state associated with the tracked object. For example, the updating component 112 can initially determine that the tracked object is in a first state between time T(0) and time T(l) because the confidence score 602 does not satisfy (e.g., is less than) a first threshold 606(1). In some examples, the first state can be associated with the tracked object not yet being verified as an actual object (e.g., a pre-tracked object). The updating component 112 can then determine that the tracked object switches from the first state to a second state at time T(l) based on the confidence score 602 satisfying (e.g., being equal to or greater than) the first threshold 606(1). In some examples, the second state can be associated with confirming that the tracked object is an actual object and not an error associated with the tracking component 102 and / or the perception system 208.

[0063] The updating component 112 can also determine that the tracked object switches from the second state to a third state at time T(5) based on the confidence score 602 not satisfying (e.g., being less than) a second threshold 604(2). As described in greater detail herein, the third state is also associated with a “pre-tracked” object, where data associated with the tracked object is no longer output by the tracking component 102. Next, the updating component 112 can determine that the tracked object switches from the third state to a fourth state at time T(6) based on the confidence score 602 not satisfying (e.g., being less than) a third threshold 604(3). As described in greater detail herein, the fourth state is associated with terminating the track associated with the object.

[0064] While Figure 6The example shows that the first threshold 606(1) is greater than the second threshold 606(2) and the third threshold 606(3), but in other examples, the first threshold 606(1) may be less than the second threshold 606(2) and / or the third threshold 606(3). Furthermore, although Figure 6 The example shows that the second threshold 606(2) is greater than the third threshold 606(3), but in other examples, the second threshold 606(2) may be less than the third threshold 606(3).

[0065] In addition, although Figure 6 The example describes using a confidence score 602 to determine the state, but in some examples, the update component 112 may use one or more additional and / or alternative techniques to determine the state. For example, the update component 112 may determine that the tracked object is switching from a first state to a second state based on a first percentage of images depicting the tracked object (e.g., based on the determination that objects detected in images at a first threshold percentage are associated with the tracked object). The first threshold percentage of images may include, but is not limited to, 50 percent, 75 percent, 90 percent, and / or any other percentage. For example, if the first threshold percentage of images is 80 percent, then the update component 112 may determine that the tracked object is switching from a first state to a second state based on the detection of the tracked object in at least four of five images.

[0066] The updating component 112 can also determine whether the tracked object is switching from a second state to a fourth state (and / or a third state) based on a second percentage of images that do not depict the tracked object (e.g., based on the determination that the tracked object is not associated with a detected object in images at a second threshold percentage). The second threshold percentage of the images can be, but is not limited to, 50 percent, 75 percent, 90 percent, and / or any other percentage. For example, if the second threshold percentage of the images is again 80 percent, then the updating component 112 can determine whether the tracked object is switching from a second state to a fourth state based on the fact that the tracked object is not detected in at least four out of five images.

[0067] refer to Figure 1 The process 100 may also include a termination component 114 of the tracking component 102 terminating the track associated with the tracked object. For example, based on the update component 112 updating the state of the tracked object to a fourth state, as described herein, the termination component 114 may terminate the track of that object. Thus, the tracking component 102 may no longer need to track the object using a new image, or it may be able to track the object using a new track if the object reappears.

[0068] The process 100 may also include a creation component 116 of the tracking component 102 creating a new track for the detected new object. For example, whenever a new object is detected by the tracking component 102 (e.g., when the association component 110 determines a detected object not associated with the tracked object and / or the update component 112 determines the object is in a first state), the creation component 116 may create a new track associated with that object. By creating a new track, the tracking component 102 may then perform the processes described herein to track the object using the new image.

[0069] The process 100 may also include tracking component 102 generating output data 118 associated with the tracked object. As described herein, the output data 118 for the tracked object may include identifiers associated with the object (e.g., numeric identifiers, alphabetic identifiers, alphanumeric identifiers, and / or any other type of identifier), the object's location (e.g., an updated bounding shape associated with the object), the object's associated classification (e.g., vehicle, pedestrian, sign, structure, road, etc.), the object's associated confidence score, and / or any other information. In some examples, the output data 118 includes information associated with one or more (e.g., all) objects associated with a second state. In some examples, the output data 118 includes information associated with one or more (e.g., all) objects associated with one or more other states.

[0070] In some examples, tracking component 102 can continue to perform the processes described herein to track the object as new sensor data 206 is generated and / or received. Furthermore, although Figures 2-6 The example describes tracking a single object 304 on an image, but in other examples, the tracking component 102 can perform a similar process to track any number of objects.

[0071] The process 100 may also include a collision component 120 configured to determine a TTC estimate associated with the tracked object. For example, and as shown, the collision component 120 may include a TTC component 122 that processes object data 106, feature point data 108, and / or data output by the tracking component 102 to determine the TTC estimate of the tracked object. In some examples, to determine the TTC estimate of the tracked object, the TTC component 122 may perform one or more of the processes described herein with respect to the tracking component 102 to determine a final scalar change associated with the tracked object (and / or may receive data representing the final scalar change from the tracking component 102). The TTC component 122 may also determine a time baseline associated with the image used to determine the final scalar change. For example, the TTC component 122 may determine the time baseline as the time difference between the times the images were generated. For example, and using... Figures 2-5BFor example, the TTC component 122 can determine a time baseline as a time difference between a time at which the image 204 was generated and a time at which the image 404 was generated. The TTC component 122 can then use the final scalar change and the time baseline to determine the TTC estimate. For example, the TTC component 122 can determine the TTC estimate by dividing the time baseline by the final scalar change.

[0072] In some examples, the TTC component 122 can perform one or more smoothing operations to determine a final TTC estimate associated with the tracked object. For example, Figure 7 Examples of using one or more smoothing techniques to determine a final TTC estimate associated with a tracked object are illustrated in accordance with some embodiments of the present disclosure. In Figure 7 In examples of determining a final TTC estimate, the TTC component 122 can perform the processes described herein to determine a TTC estimate 702(5) for the tracked object at a particular time. The TTC component 122 can then use the TTC estimates 702(2)-(5) associated with the smoothing window duration 704(2) for the TTC estimate 702(5) at the particular time to determine a final TTC estimate for the tracked object at the particular time. In some examples, the TTC component 122 determines the final TTC estimate as an average of the TTC estimates 702(2)-(5), a median of the TTC estimates 702(2)-(5), a mode of the TTC estimates 702(2)-(5), and / or using one or more additional and / or alternative techniques. In some embodiments, weighting can be used, such as weighting more recent TTC estimates 702 higher than prior TTC estimates 702.

[0073] For example, the TTC component 122 can determine a time baseline as a time difference between a time at which the image 204 was generated and a time at which the image 404 was generated. The TTC component 122 can then use the final scalar change and the time baseline to determine the TTC estimate. For example, the TTC component 122 can determine the TTC estimate by dividing the time baseline by the final scalar change.

[0074] In some embodiments, the TTC component 122 may initially validate the TTC estimate 702 associated with the object before using it in smoothing calculations. For example, Figure 8 Examples of validating TTC estimates according to some embodiments of this disclosure are shown. As shown, TTC component 122 may store object state history information 802 associated with a tracked object. In some embodiments, when tracking component 102 associates a detected object with a tracked object and / or when collision component 120 determines a TTC estimate associated with a tracked object, TTC component 122 generates and / or stores object state information 804(1)-(0) (also referred to as "object state information 804") using one or more processes described herein. For example, TTC component 122 may generate and / or store object state information 804 for one or more images (e.g., each image) for associating a detected object with a tracked object and / or determining a TTC estimate for a tracked object. In some examples, object state information 804 may include, but is not limited to, the bounding shape associated with the tracked object, the TTC estimate of the tracked object, whether the TTC estimate is valid, the final scalar change associated with the tracked object, the final translation associated with the tracked object, and / or any other information.

[0075] When new object state information 804 is generated, TTC component 122 can perform TTC verification 806 to determine whether the TTC estimate associated with object state information 802 is valid. In some examples, to perform TTC verification 806, and at box B808, prediction component 122 can initially predict the bounding shape using the final scalar change and the previous bounding shape associated with the object. For example, and referring to... Figures 2-5B For example, prediction component 122 can perform at least about Figures 2-5B The process is described in order to predict the bounding shape 416 using the final scalar change and the bounding shape 302 associated with the object 304.

[0076] Then, prediction component 122 can, in box B810, determine whether the IoU is greater than a threshold. For example, prediction component 122 can perform the procedure described herein regarding at least Figures 2-5Bthe process to determine the IoU using the predicted bounding shape 416 and the determined bounding shape 502(1) associated with the object 304. The prediction component 122 can then determine whether the IoU is greater than a threshold. If, at block B810, the prediction component 122 determines that the IoU is greater than the threshold, then at block B812, the prediction component 122 can determine that the TTC estimate is valid. In some examples, the prediction component 122 can make this determination because the final scalar change used to determine the TTC estimate is verified to be accurate. However, if, at block B814, the prediction component 122 determines that the IoU is less than or equal to the threshold, then at block B814, the prediction component 122 can determine that the TTC estimate is invalid. In some examples, the prediction component 122 can make this determination because the final scalar change is not verified to be accurate.

[0077] In some instances, and as further shown in the example of Figure 8 At block B816, the prediction component 122 can add the TTC estimate to the smoothing array based on the TTC estimate being valid, in some examples. For example, the TTC component 122 can use the valid one or more TTC estimates (e.g., all TTC estimates) while not using the invalid one or more TTC estimates (e.g., all TTC estimates) in determining the final TTC estimate using the smoothing window duration. This can increase the accuracy of the final TTC estimate because TTC estimates that are verified to be valid are used in the calculation.

[0078] Referring to Figure 1 The process 100 can include the TTC component 122 generating output data 124 associated with the tracked object, in some examples. In some examples, the output data 124 can represent at least an identifier associated with the tracked object and a TTC estimate associated with the tracked object. While Figure 1 The example of shows the tracking component 102 generating output data 118 that is separate from the output data 124 generated by the collision component 120, in other examples, the tracking component 102 and the collision component 120 can output combined data that includes the output data 118 and the output data 124.

[0079] Referring now to Figure 9 and Figure 10Each block of the methods 900 and 1000 described herein comprises a computational process that can be performed using any combination of hardware, firmware, and / or software. For instance, various functions can be implemented by a processor executing stored instructions in a memory. The methods 900 and 1000 can also be embodied as computer-usable instructions stored on computer storage media. The methods 900 and 1000 can be provided by a standalone application, a service or hosted service, an add-in to another product, or a plug-in to another product, to name a few. Figures 1-5B The methods 900 and 1000 described are provided as examples. However, these methods 900 and 1000 can additionally or alternatively be performed by any of the systems or combinations of systems, including but not limited to those described herein.

[0080] Figure 9 is a flow diagram illustrating a method 900 for tracking an object using feature points according to some embodiments of the present disclosure. The method 900, at block B902, can include determining first feature points associated with an object depicted in a first image based at least in part on first image data representative of the first image. For example, the tracking component 102 can determine the first feature points 202 associated with the first image 204. In some examples, the tracking component 102 determines the first feature points 202 using one or more techniques, such as, but not limited to, Harris corners, SIFT, SURF, FAST, Oriented FAST and ORB, and / or any other technique. In some instances, the tracking component 102 determines the first feature points 202 by receiving feature point data 108 representative of the first feature points.

[0081] The method 900, at block B904, can include determining second feature points associated with a second image based at least in part on second image data representative of the second image. For example, the tracking component 102 can perform the processes described herein to determine an initial set of feature points associated with the second image 404. The tracking component 102 can then use the enclosing shape 302 associated with the first image 204 to determine a tracking shape 402 associated with the second image 404. Using the tracking shape 402, the tracking component 102 can determine the second feature points 406 from the initial set of feature points. For example, the tracking component 102 can determine the second feature points 406 as the feature points from the initial set of feature points that are located within the tracking shape 402.

[0082] The method 900, at block B906, can include determining a scalar change associated with the object based at least in part on the first feature points and the second feature points. For example, the tracking component 102 can use the first feature points 202 and the second feature points 406 to determine a scalar change. As described herein, and in some examples, the tracking component 102 can determine the scalar change by initially determining a first distance between pairs of the first feature points 202 and a second distance between pairs of the second feature points 406. The tracking component 102 can then determine a difference between the first distance and the second distance. Using the difference, the tracking component 102 can determine a scalar change associated with the pairs of feature points. The tracking component 102 can then use the scalar change to determine a final scalar change for the object.

[0083] The method 900, at block B908, can include determining that the object is also depicted in the second image based at least in part on the scalar change. For example, the tracking component 102 can use the scalar change to determine that a detected object depicted in the second image is the object associated with the first image (e.g., the tracked object). In some examples, to make the determination, the tracking component 102 can use the enclosing shape 302, the scalar change, and the translation to determine an enclosing shape 416. The tracking component 102 can then compare the enclosing shape 416 to the enclosing shape 502 associated with the detected object. Based on the comparison, the tracking component 102 can determine a score associated with the detected object and use the score to determine that the detected object is the tracked object from the first image.

[0084] Reference is now made to Figure 10 , Figure 10 is a flowchart showing a method 1000 for determining a TTC estimate associated with an object using feature points, according to some embodiments of the present disclosure. The method 1000, at block B1002, can include determining first feature points associated with an object depicted in a first image based at least in part on first image data representative of the first image. For example, the collision component 120 can determine the first feature points 202 associated with the first image 204. In some examples, the collision component 120 determines the first feature points 202 using one or more techniques, such as, but not limited to, Harris corners, SIFT, SURF, FAST, Oriented FAST and ORB, and / or any other technique. In some examples, the collision component 120 determines the first feature points 202 by receiving feature point data 108 representative of the first feature points.

[0085] The method 1000, at block B1004, can include determining, based at least in part on second image data representative of a second image, second feature points associated with an object depicted in the second image. For example, the collision component 120 can perform the processes described herein to determine an initial set of feature points associated with the second image 404. The collision component 120 can then use the enclosing shape 302 associated with the first image 204 to determine a tracking shape 402 associated with the second image 404. Using the tracking shape 402, the collision component 120 can determine the second feature points 406 from the initial set of feature points. For example, the collision component 120 can determine the second feature points 406 as the feature points from the initial set of feature points that are located within the tracking shape 402.

[0086] The method 1000, at block B1006, can include determining, based at least in part on the first feature points and the second feature points, a scalar change associated with the object. For example, the collision component 120 can use the first feature points 202 and the second feature points 406 to determine a scalar change. As described herein, and in some examples, the collision component 120 can determine the scalar change by initially determining a first distance between pairs of the first feature points 202 and a second distance between pairs of the second feature points 406. The collision component 120 can then determine a difference between the first distance and the second distance. Using the difference, the collision component 120 can determine a scalar change associated with the pairs of feature points. The collision component 120 can then use the scalar change to determine a final scalar change for the object.

[0087] The method 1000, at block B1008, can include determining, based at least in part on the scalar change, a time-to-collision estimate associated with the object. For example, the collision component 120 can use the scalar change to determine a TTC estimate associated with the object 304. As described herein, the collision component 120 can use the scalar change and a time baseline to determine the TTC estimate. In some examples, the collision component 120 can further perform one or more smoothing operations to determine the TTC estimate.

[0088] Example autonomous vehicle

[0089] Figure 11Ais a diagram of an example autonomous vehicle 1100 according to some embodiments of the present disclosure. Autonomous vehicle 1100 (alternatively referred to herein as “vehicle 1100”) can include, but is not limited to, a passenger vehicle such as a car, truck, bus, first responder vehicle, shuttle, electric or motorized bicycle, motorcycle, fire vehicle, police vehicle, ambulance, boat, construction vehicle, underwater vessel, robotic vehicle, drone, airplane, vehicle connected to a trailer (e.g., a semi-trailer truck for hauling cargo), and / or another type of vehicle (e.g., a self-driving and / or a vehicle that accommodates one or more passengers). Autonomous vehicles are often described in terms of levels of automation as defined by a department of the United States Department of Transportation, the National Highway Traffic Safety Administration (NHTSA), and the Society of Automotive Engineers (SAE) “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles” (Standard No. J3016-201806 published June 15, 2018, Standard No. J3016-201609 published September 30, 2016, and previous and future versions of this standard). Vehicle 1100 can be capable of implementing functionality that complies with one or more of Levels 3-5 of autonomous driving. Vehicle 1100 can be capable of implementing functionality that complies with one or more of Levels 1-5 of autonomous driving. For example, depending on the embodiment, vehicle 1100 can be capable of implementing driver assistance (Level 1), partial automation (Level 2), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5). As used herein, the term “autonomous” can include any and / or all types of autonomy of vehicle 1100 or other machines, such as fully autonomous, highly autonomous, conditionally autonomous, partially autonomous, providing assistance autonomous, semi-autonomous, primarily autonomous, or other designation.

[0090] Vehicle 1100 can include components such as a chassis, a body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components of a vehicle. Vehicle 1100 can include a propulsion system 1150, such as an internal combustion engine, a hybrid power plant, an all-electric motor, and / or another type of propulsion system. Propulsion system 1150 can be connected to a drivetrain of vehicle 1100 that can include a transmission in order to effectuate propulsion of vehicle 1100. Propulsion system 1150 can be controlled in response to receiving a signal from a throttle / accelerator 1152.

[0091] A steering system 1154, which can include a steering wheel, can be used to steer the vehicle 1100 (e.g., along a desired path or route) while the propulsion system 1150 is operating (e.g., while the vehicle is in motion). The steering system 1154 can receive signals from a steering actuator 1156. For full automation (Level 5) functionality, the steering wheel can be optional.

[0092] A braking sensor system 1146 can be used to operate the vehicle brakes in response to receiving signals from a braking actuator 1148 and / or a braking sensor.

[0093] One or more controllers 1136, which can include one or more system on a chip (SoC) 1104 Figure 11C ) and / or one or more GPUs, can provide signals (e.g., representing commands) to one or more components and / or systems of the vehicle 1100. For example, the one or more controllers can send signals to operate the vehicle brakes via one or more braking actuators 1148, to operate the steering system 1154 via one or more steering actuators 1156, to operate the propulsion system 1150 via one or more throttle / accelerator 1152. The one or more controllers 1136 can include one or more on-board (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operational commands (e.g., signals representing commands) to enable autonomous driving and / or to assist a human driver in driving the vehicle 1100. The one or more controllers 1136 can include a first controller 1136 for autonomous driving functionality, a second controller 1136 for functional safety functionality, a third controller 1136 for artificial intelligence functionality (e.g., computer vision), a fourth controller 1136 for infotainment functionality, a fifth controller 1136 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 1136 can handle two or more of the above functionalities, two or more controllers 1136 can handle a single functionality, and / or any combination thereof.

[0094] One or more controllers 1136 can provide signals for controlling one or more components and / or systems of the vehicle 1100 in response to sensor data (e.g., sensor inputs) received from one or more sensors. The sensor data can be received from, for example and without limitation, a global navigation satellite system (“GNSS”) sensor 1158 (e.g., a global positioning system sensor), a RADAR sensor 1160, an ultrasonic sensor 1162, a LIDAR sensor 1164, an inertial measurement unit (IMU) sensor 1166 (e.g., an accelerometer, a gyroscope, a magnetic compass, a magnetometer, etc.), a microphone 1196, a stereo camera 1168, a wide-angle camera 1170 (e.g., a fisheye camera), an infrared camera 1172, a surround camera 1174 (e.g., a 360-degree camera), a long and / or medium range camera 1198, a speed sensor 1144 (e.g., for measuring the speed of the vehicle 1100), a vibration sensor 1142, a steering sensor 1140, a brake sensor (e.g., as part of a brake sensor system 1146), and / or other sensor types.

[0095] One or more of the controllers 1136 can receive inputs (e.g., represented by input data) from the instrument cluster 1132 of the vehicle 1100 and provide outputs (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display 1134, an audible annunciator, a speaker, and / or via other components of the vehicle 1100. These outputs can include information such as vehicle speed, velocity, time, map data (e.g., a high-definition (“HD”) map 1122 of the controller 1136), location data (e.g., a location of the vehicle 1100, e.g., on a map), direction, locations of other vehicles (e.g., an occupancy grid), information about objects and object states as perceived by the controller 1136, and / or the like. For example, the HMI display 1134 can display information about the presence of one or more objects (e.g., a street sign, a warning sign, a traffic light change, etc.) and / or information about a driving maneuver that the vehicle has made, is making, or will make (e.g., now change lanes, exit 34B in two miles, etc.). Figure 11C

[0096] ​The vehicle 1100 also includes a network interface 1124 that can communicate over one or more networks using one or more wireless antennas 1126 and / or modems. For example, the network interface 1124 can be capable of communicating over Long Term Evolution (“LTE”), Wideband Code Division Multiple Access (“WCDMA”), Universal Mobile Telecommunications System (“UMTS”), Global System for Mobile communications (“GSM”), IMT-CDMA Multi-Carrier (“CDMA2000”), and / or the like. The one or more wireless antennas 1126 can also enable communication between objects (e.g., vehicles, mobile devices, and / or the like) in an implementation environment using one or more local area networks such as Bluetooth, Bluetooth Low Energy (“LE”), Z-Wave, ZigBee, and / or the like and / or one or more low power wide area networks (“LPWANs”) such as LoRaWAN, SigFox, and / or the like.

[0097] Figure 11B For example autonomous vehicle 1100 for Figure 11A An example of camera locations and fields of view for the example autonomous vehicle 1100 for

[0098] Camera types for the cameras can include, but are not limited to, digital cameras that can be suitable for use with components and / or systems of the vehicle 1100. The cameras can operate at Automotive Safety Integrity Level (ASIL) B and / or at another ASIL. The camera types can have any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, and / or the like, depending on the embodiment. The cameras can be capable of using a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, a color filter array can include a Red-White-White-White (RCCC) color filter array, a Red-White-White-Blue (RCCB) color filter array, a Red-Blue-Green-White (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, clear pixel cameras, such as cameras with a

[0099] In some examples, one or more of the cameras can be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function monocular camera can be installed to provide functions including lane departure warning, traffic sign assist, and intelligent headlamp control. One or more of the cameras (e.g., all of the cameras) can simultaneously record and provide image data (e.g., video).

[0100] One or more of the cameras can be mounted in mounting assemblies such as custom designed (three-dimensional (3D) printed) assemblies to cut off stray light and reflections from within the car (e.g., reflections from the dashboard reflected in the windshield mirror) that can interfere with the image data capture capabilities of the cameras. With respect to wing mirror mounting assemblies, the wing mirror assemblies can be custom 3D printed such that the camera mounting plates match the shape of the wing mirrors. In some examples, one or more cameras can be integrated into the wing mirrors. For side view cameras, one or more cameras can also be integrated into the four pillars at each corner of the cab.

[0101] Cameras with fields of view that include the portion of the environment in front of the vehicle 1100 (e.g., front-facing cameras) can be used for surround view to help identify the forward path and obstacles, and to assist in providing information critical to generating an occupancy grid and / or determining a preferred vehicle path with the help of one or more controllers 1136 and / or control SoCs. Front-facing cameras can be used to perform many of the same ADAS functions as LIDAR, including emergency braking, pedestrian detection, and collision avoidance. Front-facing cameras can also be used for ADAS functions and systems including lane departure warning (“LDW”), adaptive cruise control (“ACC”), and / or other functions such as traffic sign recognition.

[0102] A wide variety of cameras can be used in the front-facing configuration, including, for example, monocular camera platforms including complementary metal-oxide-semiconductor (CMOS) color imagers. Another example can be a wide-angle camera 1170, which can be used to perceive objects (e.g., pedestrians, intersection traffic, or bicycles) entering the field of view from the periphery. Although Figure 11B Although only one wide-angle camera is illustrated in FIG. 11, there can be any number (including zero) of wide-angle cameras 1170 on the vehicle 1100. In addition, any number of long-range cameras 1198 (e.g., pairs of long-view stereo cameras) can be used for depth-based object detection, especially for objects for which a neural network has not been trained. Long-range cameras 1198 can also be used for object detection and classification and basic object tracking.

[0103] Any number of stereo cameras 1168 can also be included in the front-facing configuration. In at least one embodiment, one or more of the stereo cameras 1168 can include an integrated control unit that includes a scalable processing unit that can provide a multi-core microprocessor with an integrated Controller Area Network ("CAN") or Ethernet interface and a field programmable gate array ("FPGA") on a single chip. Such a unit can be used to generate a 3D map of the vehicle's environment, including distance estimates for all points in the image. Alternative stereo cameras 1168 can include compact stereo vision sensors that can include two camera lenses (one on the left and one on the right) and an image processing chip that can measure the distance from the vehicle to a target object and use the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 1168 can be used in addition to or instead of those described herein.

[0104] Cameras with fields of view that include portions of the environment to the side of the vehicle 1100 (e.g., side-view cameras) can be used for surround view, providing information used to create and update the occupancy grid and to generate side-crash collision warnings. For example, surround cameras 1174 (e.g., four surround cameras 1174 as shown in FIG. 17) can be placed on the vehicle 1100. The surround cameras 1174 can include wide-view cameras 1170, fisheye cameras, 360-degree cameras, and / or the like. In one example, four fisheye cameras can be placed on the front, back, and sides of the vehicle. In an alternative arrangement, the vehicle can use three surround cameras 1174 (e.g., left, right, and rear) and can utilize one or more other cameras (e.g., a forward-facing camera) as a fourth surround view camera. Figure 11B

[0105] Cameras with fields of view that include portions of the environment to the rear of the vehicle 1100 (e.g., rear-view cameras) can be used to assist with parking, surround view, rear collision warnings, and to create and update the occupancy grid. A wide variety of cameras can be used, including but not limited to cameras that are also suitable as front-facing cameras (e.g., long- and / or mid-range cameras 1198, stereo cameras 1168, infrared cameras 1172, etc.) as described herein.

[0106] Figure 11C For use in accordance with some embodiments of the present disclosure Figure 11A ​FIG. 1 illustrates an example system architecture of an example autonomous vehicle 1100. It should be understood that this arrangement and other arrangements described herein are set forth merely as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, groupings of functions, etc.) can be used in addition to or instead of those shown, and some elements can be wholly omitted. Further, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in conjunction with other components, and in any suitable combinations and locations. Various functions described herein as being performed by an entity can be implemented in hardware, firmware, and / or software. For instance, various functions can be implemented by a processor executing instructions stored in a memory.

[0107] Figure 11C Each of the components, features, and systems of the vehicle 1100 are illustrated as being connected via a bus 1102. The bus 1102 can include a controller area network (CAN) data interface (alternatively referred to herein as a "CAN bus"). The CAN can be a network within the vehicle 1100 that is used to assist in controlling various features and functions of the vehicle 1100, such as the actuation of brakes, acceleration, braking, steering, windshield wipers, etc. The CAN bus can be configured to have tens or even hundreds of nodes, each with its own unique identifier (e.g., CAN ID). The CAN bus can be read to find steering wheel angle, ground speed, revolutions per minute (RPM) of the engine, button positions, and / or other vehicle status indicators. The CAN bus can be ASIL B compliant.

[0108] Although the bus 1102 is described herein as a CAN bus, this is not intended to be limiting. For example, FlexRay and / or Ethernet can be used in addition to or instead of the CAN bus. Further, although the bus 1102 is represented with a single line, this is not intended to be limiting. For example, there can be any number of buses 1102, which can include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses that use different protocols. In some examples, two or more buses 1102 can be used to perform different functions, and / or can be used for redundancy. For example, a first bus 1102 can be used for collision avoidance functions, and a second bus 1102 can be used for drive control. In any example, each bus 1102 can be in communication with any component of the vehicle 1100, and two or more buses 1102 can be in communication with the same components. In some examples, each SoC 1104, each controller 1136, and / or each computer within the vehicle can have access to the same input data (e.g., inputs from sensors of the vehicle 1100), and can be connected to a common bus, such as a CAN bus.

[0109] The vehicle 1100 can include one or more controllers 1136, such as those described herein with respect to Figure 11A The controllers 1136 can be used for a wide variety of functions. The controllers 1136 can be coupled to any of the other various components and systems of the vehicle 1100 and can be used for control of the vehicle 1100, artificial intelligence of the vehicle 1100, infotainment for the vehicle 1100, and / or the like.

[0110] The vehicle 1100 can include one or more system on chips (SoCs) 1104. The SoCs 1104 can include CPUs 1106, GPUs 1108, processors 1110, caches 1112, accelerators 1114, data stores 1116, and / or other components and features not illustrated. The SoCs 1104 can be used to control the vehicle 1100 in a wide variety of platforms and systems. For example, one or more SoCs 1104 can be used in a system (such as a system of the vehicle 1100) in conjunction with an HD map 1122 that can obtain map refreshes and / or updates from one or more servers (such as the one or more servers 1178) via a network interface 1124. Figure 11D

[0111] The CPU 1106 can include a CPU cluster or CPU complex (alternatively referred to herein as a “CCPLEX”). The CPU 1106 can include multiple cores and / or L2 caches. For example, in some embodiments, the CPU 1106 can include eight cores in a coherent multi-processor configuration. In some embodiments, the CPU 1106 can include four dual-core clusters with each cluster having a dedicated L2 cache (such as a 2 MB L2 cache). The CPU 1106 (e.g., the CCPLEX) can be configured to support simultaneous cluster operation such that any combination of clusters of the CPU 1106 can be active at any given time.

[0112] ​The CPU 1106 can implement power management capabilities including one or more of the following features: individual hardware blocks can be automatically clock-gated when idle to save dynamic power; each core clock can be gated when the core is not actively executing instructions due to execution of WFI / WFE instructions; each core can be independently power-gated; each core cluster can be independently clock-gated when all cores are clock-gated or power-gated; and / or each core cluster can be independently power-gated when all cores are power-gated. The CPU 1106 can further implement enhanced algorithms for managing power states, where the allowed power states and desired wake-up times are specified and the hardware / microcode determines the optimal power state for the cores, clusters, and CCPLEX to enter. The processing cores can support a simplified power state entry sequence in software, with the work offloaded to microcode.

[0113] The GPU 1108 can include an integrated GPU (alternatively referred to herein as an “iGPU”). The GPU 1108 can be programmable and efficient for parallel workloads. In some examples, the GPU 1108 can use an enhanced tensor instruction set. The GPU 1108 can include one or more streaming microprocessors, where each streaming microprocessor can include an LI cache (e.g., an LI cache having at least 96 KB of storage capacity), and two or more of the streaming microprocessors can share an L2 cache (e.g., an L2 cache having 512 KB of storage capacity). In some embodiments, the GPU 1108 can include at least eight streaming microprocessors. The GPU 1108 can use a compute application programming interface (API). Further, the GPU 1108 can use one or more parallel computing platforms and / or programming models (e.g., NVIDIA’s CUDA).

[0114] In the case of automotive and embedded uses, the GPU 1108 can be power-optimized for best performance. For example, the GPU 1108 can be fabricated on a fin field-effect transistor (FinFET) for lower power consumption. However, this is not intended to be limiting, and the GPU 1108 can be fabricated using other semiconductor manufacturing processes. Each streaming microprocessor can incorporate several mixed-precision processing cores divided into multiple blocks. For example, and without limitation, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA Tensor Cores for deep learning matrix arithmetic, an L0 instruction cache, a thread warp scheduler, a dispatch unit, and / or a 64 KB register file. Further, the streaming microprocessor can include independent parallel integer and floating point data paths to exploit the mix of computation and address computation for efficient execution of workloads. The streaming microprocessor can include independent thread scheduling capabilities to allow for finer-grain synchronization and cooperation between parallel threads. The streaming microprocessor can include a combined LI data cache and shared memory unit to improve performance while simplifying programming.

[0115] The GPU 1108 can include a high bandwidth memory (HBM) and / or a 16 GB HBM2 memory subsystem that provides approximately 900 GB / s of peak memory bandwidth in some examples. In some examples, in addition to or alternatively from HBM memory, synchronous graphics random access memory (SGRAM) can be used, such as fifth generation graphics double data rate synchronous random access memory (GDDR5).

[0116] The GPU 1108 can include a unified memory technology that includes an access counter to allow memory pages to be migrated more precisely to the processors that access them most frequently, improving efficiency of memory ranges shared between processors. In some examples, address translation services (ATS) support can be used to allow the GPU 1108 to access CPU 1106 page tables directly. In such examples, when the GPU 1108 memory management unit (MMU) experiences a miss, an address translation request can be transmitted to the CPU 1106. In response, the CPU 1106 can look up the virtual-to-physical mapping for the address in its page tables and transmit the translation back to the GPU 1108. In this way, the unified memory technology can allow a single unified virtual address space for memory of both the CPU 1106 and the GPU 1108, simplifying GPU 1108 programming and porting applications to the GPU 1108.

[0117] In addition, GPU 1108 can include an access counter that can track how often GPU 1108 accesses memory of other processors. The access counter can help ensure that memory pages are moved to the physical memory of the processor that most frequently accesses those pages.

[0118] SoC 1104 can include any number of caches 1112, including those described herein. For example, caches 1112 can include an L3 cache that is available to both CPU 1106 and GPU 1108 (e.g., connected to both CPU 1106 and GPU 1108). Caches 1112 can include a write-back cache that can track the state of a line, for example, by using a cache coherency protocol (e.g., MEI, MESI, MSI, etc.). Depending on the embodiment, the L3 cache can include 4MB or more, although smaller cache sizes can also be used.

[0119] SoC 1104 can include an arithmetic logic unit (ALU) that can be utilized in processing to perform any of a variety of tasks or operations with respect to vehicle 1100, such as processing a DNN. In addition, SoC 1104 can include a floating point unit (FPU) (or other mathematical co-processor or digital co-processor type) for performing mathematical operations within the system. For example, SoC 104 can include one or more FPUs integrated as execution units within CPU 1106 and / or GPU 1108.

[0120] SoC 1104 can include one or more accelerators 1114 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, SoC 1104 can include a hardware accelerator cluster that can include optimized hardware accelerators and / or a large on-chip memory. The large on-chip memory (e.g., 4MB SRAM) can enable the hardware accelerator cluster to accelerate neural networks and other computations. The hardware accelerator cluster can be used to supplement GPU 1108 and offload some of the tasks of GPU 1108 (e.g., freeing up more cycles of GPU 1108 for performing other tasks). As one example, accelerators 1114 can be used for targeted workloads (e.g., perception, convolutional neural networks (CNNs), etc.) that are stable enough to accelerate easily. As used herein, the term “CNN” can include all types of CNNs, including region-based or region convolutional neural networks (RCNNs) and fast RCNNs (e.g., for object detection).

[0121] The accelerators 1114 (e.g., a cluster of hardware accelerators) can include a deep learning accelerator (DLA). The DLA can include one or more tensor processing units (TPUs) that can be configured to provide an additional 100 billion operations per second for deep learning applications and inferencing. The TPU can be an accelerator that is configured to perform image processing functions (e.g., for CNNs, RCNNs, etc.) and is optimized for performing image processing functions. The DLA can be further optimized for a specific set of neural network types and floating point operations and inferencing. The design of the DLA can provide higher performance per mm than general purpose GPUs and far exceeds the performance of CPUs. The TPU can perform several functions including single instance convolution functions, support for INT8, INT16, and FP16 data types for both features and weights, for example, and post-processor functions.

[0122] The DLA can perform neural networks, especially CNNs, on processed or unprocessed data for any of a wide variety of functions, such as and not by way of limitation: CNNs for object recognition and detection using data from camera sensors; CNNs for distance estimation using data from camera sensors; CNNs for emergency vehicle detection and identification and detection using data from microphones; CNNs for facial recognition and vehicle owner identification using data from camera sensors; and / or CNNs for safety and / or safety related events.

[0123] The DLA can perform any of the functions of the GPU 1108, and by using an inferencing accelerator, the designer can target the DLA or the GPU 1108 for any function, for example. For example, the designer can focus the processing and floating point operations of CNNs on the DLA and leave other functions to the GPU 1108 and / or other accelerators 1114.

[0124] The accelerators 1114 (e.g., a cluster of hardware accelerators) can include a programmable vision accelerator (PVA), which can be alternatively referred to herein as a computer vision accelerator. The PVA can be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. The PVA can provide a balance between performance and flexibility. For example, each PVA can include any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors, for example and not by way of limitation.

[0125] The RISC cores can interact with image sensors (e.g., image sensors of any of the cameras described herein), image signal processors, and / or the like. Each of these RISC cores can include any number of memories. Depending on the embodiment, the RISC cores can use any of several protocols. In some examples, the RISC cores can execute a real-time operating system (RTOS). The RISC cores can be implemented using one or more integrated circuit devices, application specific integrated circuits (ASICs), and / or memory devices. For example, the RISC cores can include instruction caches and / or tightly coupled RAM.

[0126] The DMA can enable components of the PVA to access system memory independently of the CPU 1106. The DMA can support any number of features to provide optimization to the PVA, including but not limited to supporting multi-dimensional addressing and / or circular addressing. In some examples, the DMA can support addressing up to six or more dimensions, which can include block width, block height, block depth, horizontal block step, vertical block step, and / or depth step.

[0127] The vector processor can be a programmable processor that can be designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing capabilities. In some examples, the PVA can include a PVA core and two vector processing subsystem partitions. The PVA core can include a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. The vector processing subsystems can operate as the main processing engines of the PVA and can include a vector processing unit (VPU), an instruction cache, and / or a vector memory (e.g., VMEM). The VPU core can include a digital signal processor such as, for example, a single instruction multiple data (SIMD), very long instruction word (VLIW) digital signal processor. The combination of SIMD and VLIW can enhance throughput and speed.

[0128] Each of the vector processors can include an instruction cache and can be coupled to a dedicated memory. As a result, in some examples, each of the vector processors can be configured to execute independently of the other vector processors. In other examples, the vector processors included in a particular PVA can be configured to employ data parallelization. For example, in some embodiments, multiple vector processors included in a single PVA can execute the same computer vision algorithm, but on different regions of an image. In other examples, the vector processors included in a particular PVA can execute different computer vision algorithms on the same image simultaneously, or even different algorithms on a sequence of images or portions of an image. Any number of PVAs can be included in a hardware accelerator cluster, and any number of vector processors can be included in each of those PVAs, among other things. Furthermore, the PVAs can include additional error-correcting code (ECC) memory to enhance overall system security.

[0129] The accelerator 1114 (e.g., hardware accelerator cluster) can include on-chip computer vision networks and SRAM to provide high bandwidth, low latency SRAM for the accelerator 1114. In some examples, the on-chip memory can include at least 4 MB of SRAM composed of, for example and without limitation, eight field-programmable memory blocks, which can be accessed by both the PVA and the DLA. Each pair of memory blocks can include an advanced peripheral bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory can be used. The PVA and the DLA can access the memory via a backbone that provides high-speed memory access to the PVA and the DLA. The backbone can include on-chip computer vision networks that interconnect the PVA and the DLA to the memory (e.g., using APB).

[0130] The on-chip computer vision networks can include an interface that determines that both the PVA and the DLA provide ready and valid signals before transmitting any control signals / addresses / data. Such an interface can provide separate phases and separate channels for transmitting control signals / addresses / data, as well as burst communications for continuous data transmission. This type of interface can comply with ISO 26262 or IEC 61508 standards, but other standards and protocols can also be used.

[0131] In some examples, the SoC 1104 can include a real-time ray tracing hardware accelerator, such as described in U.S. Patent Application No. 16 / 101,232, filed August 10, 2018. The real-time ray tracing hardware accelerator can be used to quickly and efficiently determine locations and extents of objects (e.g., within a world model) in order to generate real-time visualizations simulations for RADAR signal interpretation, for sound propagation synthesis and / or analysis, for SONAR system simulation, for general wave propagation simulation, for comparison to LIDAR data for purposes of localization and / or other functionality, and / or for other uses. In some embodiments, one or more tree traversal units (TTUs) can be used to perform one or more ray tracing related operations.

[0132] The accelerator 1114 (e.g., a hardware accelerator cluster) has a wide range of uses for autonomous driving. The PVA can be a programmable vision accelerator that can be used for key processing stages in ADAS and autonomous vehicles. The capabilities of the PVA are a good match for algorithm domains that require predictable processing, low power, and low latency. In other words, the PVA performs well on semi-dense or dense regular computations, and even on small data sets that require predictable runtimes with low latency and low power. Thus, in the context of a platform for autonomous vehicles, the PVA is designed to run classical computer vision algorithms because they are effective at object detection and integer math operations.

[0133] For example, according to one embodiment of the technology, the PVA is used to perform computer stereo vision. In some examples, a semi-global matching based algorithm can be used, although this is not intended to be limiting. Many applications for level 3-5 autonomous driving require instant motion estimation / stereo matching (e.g., structure from motion, pedestrian recognition, lane detection, etc.). The PVA can perform computer stereo vision functions on input from two monocular cameras.

[0134] In some examples, the PVA can be used to perform dense optical flow. According to a process, raw RADAR data is processed (e.g., using a 4D fast Fourier transform) to provide processed RADAR. In other examples, the PVA is used for time-of-flight depth processing, such as by processing raw time-of-flight data to provide processed time-of-flight data.

[0135] The DLA can be used to run any type of network to enhance control and driving safety, including, for example, a neural network that outputs a confidence metric for each object detection. Such a confidence value can be interpreted as a probability, or as providing a relative "weight" for each detection compared to other detections. The confidence value enables the system to make further decisions about which detections should be considered true positive detections and not false positive detections. For example, the system can set a threshold for confidence, and only consider detections that exceed the threshold as true positive detections. In an automatic emergency braking (AEB) system, false positive detections would cause the vehicle to automatically perform an emergency brake, which is obviously undesirable. Thus, only the most confident detections should be considered a trigger for AEB. The DLA can run a neural network for regression of a confidence value. The neural network can take as its input at least some subset of parameters, such as a bounding box dimension, a ground plane estimate obtained (e.g., from another subsystem), inertial measurement unit (IMU) sensor 1166 outputs related to vehicle 1100 orientation, distance, 3D position estimates of objects obtained from the neural network and / or other sensors (e.g., LIDAR sensor 1164 or RADAR sensor 1160), etc.

[0136] SoC 1104 can include one or more data stores 1116 (e.g., memory). Data stores 1116 can be on-chip memory of SoC 1104, which can store neural networks to be executed on the GPU and / or DLA. In some examples, for redundancy and safety, data stores 1116 can be large enough in capacity to store multiple instances of a neural network. Data stores 1112 can include L2 or L3 cache 1112. References to data stores 1116 can include references to memory associated with PVA, DLA, and / or other accelerators 1114 as described herein.

[0137] SoC 1104 can include one or more processors 1110 (e.g., embedded processors). The processors 1110 can include a boot and power management processor, which can be a specialized processor and subsystem for handling boot power and management functions, as well as security implementation. The boot and power management processor can be part of the SoC 1104 boot sequence and can provide runtime power management services. The boot power and management processor can provide clock and voltage programming, auxiliary system low power state transitions, SoC 1104 thermal and temperature sensor management, and / or SoC 1104 power state management. Each temperature sensor can be implemented as a ring oscillator whose output frequency is proportional to temperature, and the SoC 1104 can use the ring oscillator to detect the temperature of the CPU 1106, GPU 1108, and / or accelerator 1114. If it is determined that the temperature exceeds a threshold, the boot and power management processor can enter a temperature fault routine and place the SoC 1104 in a lower power state and / or place the vehicle 1100 in a driver safe park mode (e.g., safely park the vehicle 1100).

[0138] The processors 1110 can also include a set of embedded processors that can be used as an audio processing engine. The audio processing engine can be an audio subsystem that allows for full hardware support for multi-channel audio over multiple interfaces, as well as a range of extensive and flexible audio I / O interfaces. In some examples, the audio processing engine is a specialized processor core with a digital signal processor with dedicated RAM.

[0139] The processors 1110 can also include an always-on processor engine, which can provide the necessary hardware features to support low-power sensor management and wake-up use cases. The always-on processor engine can include a processor core, tightly coupled RAM, supporting peripherals (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0140] The processors 1110 can also include a security cluster engine, which includes a specialized processor subsystem that handles security management for automotive applications. The security cluster engine can include two or more processor cores, tightly coupled RAM, supporting peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In a secure mode, the two or more cores can operate in a lockstep mode and act as a single core with comparison logic that detects any differences between their operations.

[0141] The processors 1110 can also include a real-time camera engine, which can include a specialized processor subsystem for handling real-time camera management.

[0142] The processor 1110 can further include a high dynamic range signal processor, which can include an image signal processor, which is a hardware engine that is part of the camera processing pipeline.

[0143] The processor 1110 can include a video image compositor, which can be a processing block (e.g., implemented on a microprocessor), that implements video post-processing functions needed by the video playback application to produce the final image for the player window. The video image compositor can perform lens distortion correction on the wide-angle camera 1170, the surround camera 1174, and / or on the cab-in monitor camera sensors. The cab-in monitor camera sensors are preferably monitored by a neural network running on another instance of the advanced SoC, configured to recognize cab-in events and respond accordingly. The cab-in system can perform lip reading to activate mobile phone services and place a call, dictate an email, change the vehicle destination, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. Certain functions are only available to the driver when the vehicle is operating in autonomous mode, and are disabled otherwise.

[0144] The video image compositor can include enhanced temporal noise reduction for spatial and temporal noise reduction. For example, where motion is present in the video, the noise reduction appropriately weights the spatial information, reducing the weight of information provided by neighboring frames. Where the image or portions of the image do not include motion, the temporal noise reduction performed by the video image compositor can use information from previous images to reduce noise in the current image.

[0145] The video image compositor can also be configured to perform stereo correction on input stereo lens frames. The video image compositor can further be used for user interface composition when the operating system desktop is in use and the GPU 1108 does not need to continuously render new surfaces. Even when the GPU 1108 is powered on and active, doing 3D rendering, the video image compositor can be used to offload the GPU 1108 to improve performance and responsiveness.

[0146] The SoC 1104 can further include a Mobile Industry Processor Interface (MIPI) camera serial interface for receiving video and input from cameras, a high-speed interface, and / or a video input block that can be used for camera and related pixel input functions. The SoC 1104 can further include an input / output controller that can be controlled by software and can be used to receive I / O signals that are not committed to a particular role.

[0147] The SoC 1104 can also include a wide range of peripheral device interfaces to enable communication with peripherals, audio codecs, power management, and / or other devices. The SoC 1104 can be used to process data from cameras (connected over Gigabit Multimedia Serial Link and Ethernet), sensors (e.g., LIDAR sensor 1164, RADAR sensor 1160, etc. that can be connected over Ethernet), data from the bus 1102 (e.g., speed of the vehicle 1100, steering wheel position, etc.), data from GNSS sensor 1158 (connected over Ethernet or CAN bus). The SoC 1104 can also include dedicated high performance mass storage controllers that can include their own DMA engines and that can be used to free up the CPU 1106 from routine data management tasks.

[0148] The SoC 1104 can be an end-to-end platform with a flexible architecture that spans automation levels 3-5, providing an integrated functional safety architecture for a platform that leverages and efficiently uses computer vision and ADAS technology to achieve diversity and redundancy, along with deep learning tools. The SoC 1104 can be faster, more reliable, and even more energy and space efficient than conventional systems. For example, the accelerators 1114, when combined with the CPU 1106, GPU 1108, and data storage 1116, can provide a fast and efficient platform for level 3-5 autonomous vehicles.

[0149] The technology thus provides capabilities and functionality that cannot be achieved by conventional systems. For example, computer vision algorithms can be executed on CPUs that can be configured using high-level programming languages such as the C programming language to perform a wide variety of processing algorithms across a wide variety of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to, for example, execution time and power consumption. In particular, many CPUs cannot execute complex object detection algorithms in real time, which is a requirement for on-board ADAS applications and for practical level 3-5 autonomous vehicles.

[0150] In contrast to conventional systems, by providing a CPU complex, a GPU complex, and a cluster of hardware accelerators, the technology described herein allows multiple neural networks to be executed simultaneously and / or sequentially, and the results to be combined together to achieve level 3-5 autonomous driving functionality. For example, a CNN executed on a DLA or dGPU (e.g., GPU 1120) can include text and word recognition, allowing a supercomputer to read and understand traffic signs, including signs for which a neural network has not been specifically trained. The DLA can also include a neural network that is able to recognize, interpret, and provide a semantic understanding of the sign, and pass that semantic understanding to a path planning module running on the CPU complex.

[0151] As another example, multiple neural networks can be run simultaneously as required for level 3, 4, or 5 driving. For example, a warning sign consisting of the words "Caution: flashing lights indicate icy conditions" along with electric lights can be interpreted by several neural networks independently or collectively. The sign itself can be recognized by a first deployed neural network (e.g., a trained neural network) as a traffic sign, the text "flashing lights indicate icy conditions" can be interpreted by a second deployed neural network that informs the vehicle's path planning software (preferably executing on the CPU complex) that icy conditions exist when flashing lights are detected. The flashing lights can be recognized by operating a third deployed neural network over multiple frames that informs the vehicle's path planning software of the presence (or absence) of flashing lights. All three neural networks can be run simultaneously, for example, within the DLA and / or on the GPU 1108.

[0152] In some examples, a CNN for face recognition and owner recognition can use data from the camera sensors to recognize the presence of an authorized driver and / or owner of the vehicle 1100. A processing engine always on the sensor can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and in a safe mode, disable the vehicle when the owner leaves the vehicle. In this way, the SoC 1104 provides security against theft and / or carjacking.

[0153] In another example, a CNN for emergency vehicle detection and recognition can use data from the microphones 1196 to detect and recognize emergency vehicle sirens. In contrast to conventional systems that use a general classifier to detect sirens and manually extract features, the SoC 1104 uses a CNN to classify ambient and urban sounds as well as to classify visual data. In a preferred embodiment, a CNN running on the DLA is trained to recognize the relative closing speed of an emergency vehicle (e.g., by using the Doppler effect). The CNN can also be trained to recognize emergency vehicles specific to the local area in which the vehicle is operating as recognized by the GNSS sensor 1158. Thus, for example, when operating in Europe, the CNN will seek to detect European sirens, and when in the United States, the CNN will seek to recognize sirens that are only North American. Once an emergency vehicle is detected, a control program can be used to execute an emergency vehicle safety routine to slow the vehicle, pull over to the side of the road, stop the vehicle, and / or idle the vehicle until the emergency vehicle passes, with the assistance of the ultrasonic sensors 1162.

[0154] The vehicle can include a CPU 1118 (e.g., a discrete CPU or dCPU) that can be coupled to the SoC 1104 via a high-speed interconnect (e.g., PCIe). The CPU 1118 can include, for example, an X86 processor. The CPU 1118 can be used to perform any of a wide variety of functions, including, for example, arbitrating potentially inconsistent results between ADAS sensors and the SoC 1104, and / or monitoring the status and health of the controller 1136 and / or infotainment SoC 1130.

[0155] The vehicle 1100 can include a GPU 1120 (e.g., a discrete GPU or dGPU) that can be coupled to the SoC 1104 via a high-speed interconnect (e.g., NVIDIA’s NVLINK). The GPU 1120 can provide additional artificial intelligence functionality, for example, by executing redundant and / or different neural networks, and can be used to train and / or update neural networks based at least in part on input (e.g., sensor data) from sensors of the vehicle 1100.

[0156] The vehicle 1100 can also include a network interface 1124 that can include one or more wireless antennas 1126 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). The network interface 1124 can be used to enable wireless connections over the Internet with a cloud (e.g., with the server 1178 and / or other network devices), with other vehicles, and / or with computing devices (e.g., client devices of passengers). For communication with other vehicles, a direct link can be established between the two vehicles, and / or an indirect link can be established (e.g., across a network and through the Internet). The direct link can be provided using a car-to-car communication link. The car-to-car communication link can provide the vehicle 1100 with information about vehicles that are approaching the vehicle 1100 (e.g., vehicles in front of, to the side of, and / or behind the vehicle 1100). This functionality can be part of a cooperative adaptive cruise control functionality of the vehicle 1100.

[0157] The network interface 1124 can include a SoC that provides modulation and demodulation functionality and enables the controller 1136 to communicate over a wireless network. The network interface 1124 can include a radio frequency front end for up- and down-shifting from baseband to radio frequency and from radio frequency to baseband. The frequency shifting can be performed through well-known processes and / or can use a super-heterodyne process. In some examples, the radio frequency front end functionality can be provided by a separate chip. The network interface can include wireless functionality for communicating over LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0158] The vehicle 1100 can also include a data store 1128, which can include off-chip (e.g., off-SoC 1104) storage. The data store 1128 can include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash memory, hard disks, and / or other components and / or devices that can store data for at least one bit.

[0159] The vehicle 1100 can also include a GNSS sensor 1158. The GNSS sensor 1158 (e.g., GPS, assisted GPS sensor, differential GPS (DGPS) sensor, etc.) is used to assist in mapping, perception, occupancy grid generation, and / or path planning functions. Any number of GNSS sensors 1158 can be used, including, for example and without limitation, a GPS using a USB connector with an Ethernet-to-serial (RS-232) bridge.

[0160] The vehicle 1100 can also include a RADAR sensor 1160. The RADAR sensor 1160 can be used by the vehicle 1100 for long-range vehicle detection, even in darkness and / or adverse weather conditions. The RADAR functional safety level can be ASIL B. The RADAR sensor 1160 can use the CAN and / or the bus 1102 (e.g., to transmit data generated by the RADAR sensor 1160) for control as well as access to object tracking data, in some examples, Ethernet for access to raw data. A wide variety of RADAR sensor types can be used. For example and without limitation, the RADAR sensor 1160 can be suitable for front, rear, and side RADAR use. In some examples, a pulsed Doppler RADAR sensor is used.

[0161] The RADAR sensor 1160 can include different configurations, such as long-range with narrow field of view, short-range with wide field of view, short-range side coverage, and so on. In some examples, long-range RADAR can be used for adaptive cruise control functionality. Long-range RADAR systems can provide a wide field of view (e.g., 250 m range) implemented through two or more independent scans. The RADAR sensor 1160 can help distinguish between static and moving objects, and can be used by the ADAS system for emergency brake assist and forward collision warning. The long-range RADAR sensor can include a single-station multi-mode RADAR with multiple (e.g., six or more) fixed RADAR antennas, as well as a high-speed CAN and FlexRay interface. In examples with six antennas, the central four antennas can create focused beam patterns designed to record the surroundings of the vehicle 1100 at higher speed with minimal traffic interference from adjacent lanes. The other two antennas can extend the field of view, making it possible to quickly detect vehicles entering or leaving the lane of the vehicle 1100.

[0162] As one example, a mid-range RADAR system can include a range of up to 1160 m (front) or 80 m (rear) and a field of view of up to 42 degrees (front) or 1150 degrees (rear). A short-range RADAR system can include, but is not limited to, RADAR sensors designed to be mounted at both ends of the rear bumper. When mounted at both ends of the rear bumper, such a RADAR sensor system can create two beams that continuously monitor the rear and the blind spot next to the vehicle.

[0163] A short-range RADAR system can be used in an ADAS system for blind spot detection and / or lane change assist.

[0164] The vehicle 1100 can also include ultrasonic sensors 1162. Ultrasonic sensors 1162 that can be placed on the front, rear, and / or sides of the vehicle 1100 can be used for parking assist and / or to create and update an occupancy grid. A wide variety of ultrasonic sensors 1162 can be used, and different ultrasonic sensors 1162 can be used for different detection ranges (e.g., 2.5 m, 4 m). The ultrasonic sensors 1162 can operate at an ASIL B functional safety level.

[0165] The vehicle 1100 can include LIDAR sensors 1164. The LIDAR sensors 1164 can be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LIDAR sensors 1164 can be at an ASIL B functional safety level. In some examples, the vehicle 1100 can include multiple LIDAR sensors 1164 (e.g., two, four, six, etc.) that can use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0166] In some examples, the LIDAR sensors 1164 can be capable of providing a list of objects and their distances for a 360-degree field of view. A commercially available LIDAR sensor 1164 can have, for example, an advertised range of approximately 1100 m, a precision of 2 cm - 3 cm, and support for a 1100 Mbps Ethernet connection. In some examples, one or more flush-mounted LIDAR sensors 1164 can be used. In such examples, the LIDAR sensors 1164 can be implemented as small devices that can be embedded into the front, rear, sides, and / or corners of the vehicle 1100. In such examples, the LIDAR sensors 1164 can provide a field of view of up to 120 degrees horizontal and 35 degrees vertical for low reflectivity objects, with a range of 200 m. Front-mounted LIDAR sensors 1164 can be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0167] In some examples, LIDAR technology such as 3D Flash LIDAR can also be used. 3D Flash LIDAR uses a flash of laser light as a source of emission to illuminate the vehicle’s surroundings up to about 200 m. The flash LIDAR unit includes a receptor that records the laser pulse transmission time and reflected light on each pixel, which in turn corresponds to the range from the vehicle to the object. Flash LIDAR can allow for the generation of highly accurate and distortion-free images of the surroundings with each laser flash. In some examples, four flash LIDAR sensors can be deployed, one on each side of the vehicle 1100. Available 3D flash LIDAR systems include solid-state 3D staring array LIDAR cameras (e.g., non-scanning LIDAR devices) that have no moving parts other than a fan. The flash LIDAR device can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture the reflected laser light in the form of 3D range point clouds and co-registered intensity data. By using flash LIDAR, and because flash LIDAR is a solid-state device with no moving parts, the LIDAR sensor 1164 can be less susceptible to motion blur, vibration, and / or jostling.

[0168] The vehicle can also include an IMU sensor 1166. In some examples, the IMU sensor 1166 can be located at the center of the rear axle of the vehicle 1100. The IMU sensor 1166 can include, for example and without limitation, an accelerometer, a magnetometer, a gyroscope, a magnetic compass, and / or other sensor types. In some examples, such as in six-axis applications, the IMU sensor 1166 can include an accelerometer and a gyroscope, while in nine-axis applications, the IMU sensor 1166 can include an accelerometer, a gyroscope, and a magnetometer.

[0169] In some embodiments, the IMU sensor 1166 can be implemented as a microelectromechanical systems (MEMS) based inertial navigation system (INS) that combines a microelectromechanical systems (MEMS) inertial sensor, a high-sensitivity GPS receiver, and advanced Kalman filtering algorithms to provide estimates of position, velocity, and attitude. As such, in some examples, the IMU sensor 1166 can enable the vehicle 1100 to estimate heading without input from a magnetic sensor by directly observing the change in velocity from GPS to the IMU sensor 1166 and correlating it. In some examples, the IMU sensor 1166 and the GNSS sensor 1158 can be combined into a single integrated unit.

[0170] The vehicle can include a microphone 1196 placed in and / or around the vehicle 1100. The microphone 1196 can be used for emergency vehicle detection and identification, among other things.

[0171] The vehicle can also include any number of camera types, including stereo cameras 1168, wide-view cameras 1170, infrared cameras 1172, surround-view cameras 1174, long and / or mid-range cameras 1198, and / or other camera types. These cameras can be used to capture image data around the entire periphery of the vehicle 1100. The types of cameras used depend on the embodiment and requirements of the vehicle 1100, and any combination of camera types can be used to provide the necessary coverage around the vehicle 1100. Further, the number of cameras can vary depending on the embodiment. For example, the vehicle can include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. As one example and not by way of limitation, the cameras can support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet. Each of the cameras is described in more detail herein with respect to Figure 11A and Figure 11B are described in more detail.

[0172] The vehicle 1100 can also include vibration sensors 1142. The vibration sensors 1142 can measure vibrations of components of the vehicle, such as axles. For example, changes in vibration can indicate changes in the road surface. In another example, when two or more vibration sensors 1142 are used, differences between the vibrations can be used to determine the friction or slip of the road surface (e.g., when there is a difference in vibration between a power driven axle and a free spinning axle).

[0173] The vehicle 1100 can include an ADAS system 1138. In some examples, the ADAS system 1138 can include a SoC. The ADAS system 1138 can include adaptive / automatic / autonomous cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keep assist (LKA), blind spot warning (BSW), rear cross-traffic warning (RCTW), collision warning system (CWS), lane centering (LC), and / or other features and functionality.

[0174] An ACC system can use RADAR sensors 1160, LIDAR sensors 1164, and / or cameras. An ACC system can include longitudinal ACC and / or lateral ACC. Longitudinal ACC monitors and controls the distance to the vehicle immediately ahead of the vehicle 1100 and automatically adjusts the vehicle speed to maintain a safe distance from the vehicle ahead. Lateral ACC performs distance keeping and, if necessary, suggests a lane change for the vehicle 1100. Lateral ACC is related to other ADAS applications such as LCA and CWS.

[0175] CACC uses information from other vehicles, which can be received from other vehicles indirectly via a wireless link or through a network connection (e.g., through the Internet) via the network interface 1124 and / or the wireless antenna 1126. Direct links can be provided by a vehicle-to-vehicle (V2V) communication link, while indirect links can be an infrastructure-to-vehicle (I2V) communication link. Generally, the V2V communication concept provides information about the immediately preceding vehicles (e.g., vehicles immediately ahead of and in the same lane as the vehicle 1100), while the I2V communication concept provides information about traffic further ahead. A CACC system can include either or both of I2V and V2V information sources. Given information about vehicles ahead of the vehicle 1100, CACC can be more reliable, and it has the potential to improve traffic flow and reduce road congestion.

[0176] FCW systems are designed to alert the driver to a hazard so that the driver can take corrective action. FCW systems use a front-facing camera and / or RADAR sensor 1160 coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as displays, speakers, and / or vibrating components. FCW systems can provide warnings in the form of, for example, sound, visual warnings, vibrations, and / or quick brake pulses.

[0177] AEB systems detect an impending forward collision with another vehicle or other object and can automatically apply the brakes if the driver does not take corrective action within specified time or distance parameters. AEB systems can use a front-facing camera and / or RADAR sensor 1160 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. When an AEB system detects a hazard, it typically first alerts the driver to take corrective action to avoid a collision, and if the driver does not take corrective action, the AEB system can automatically apply the brakes in an effort to prevent or at least mitigate the effects of a predicted collision. AEB systems can include technologies such as dynamic brake support and / or crash imminent braking.

[0178] LDW systems provide visual, audible, and / or tactile warnings such as steering wheel or seat vibrations to alert the driver when the vehicle 1100 is crossing lane markers. The LDW system is not activated when the driver indicates an intentional lane departure by activating a turn signal. LDW systems can use a front-side facing camera coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as displays, speakers, and / or vibrating components.

[0179] An LKA system is a variation of the LDW system. If the vehicle 1100 begins to leave the lane, the LKA system provides a steering input or brake to correct the vehicle 1100.

[0180] A BSW system detects and warns the driver of vehicles in the car's blind spot. The BSW system can provide visual, audible, and / or tactile alerts to indicate that merging or changing lanes is unsafe. The system can provide additional warnings when the driver uses a turn signal. The BSW system can use rear side-facing cameras and / or RADAR sensors 1160 coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as a display, speaker, and / or vibrating component.

[0181] A RCTW system can provide visual, audible, and / or tactile notifications when objects are detected outside the range of the rear-facing camera while the vehicle 1100 is backing up. Some RCTW systems include AEB to ensure that vehicle brakes are applied to avoid a collision. The RCTW system can use one or more rear-facing RADAR sensors 1160 coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as a display, speaker, and / or vibrating component.

[0182] Conventional ADAS systems can be prone to false positive results, which can annoy and distract the driver, but typically are not catastrophic because the ADAS system alerts the driver and allows the driver to decide whether the safety condition is truly present and act accordingly. However, in an autonomous vehicle 1100, in the case of conflicting results, the vehicle 1100 itself must decide whether to heed the results from the primary computer or the secondary computer (e.g., the first controller 1136 or the second controller 1136). For example, in some embodiments, the ADAS system 1138 can be a secondary and / or auxiliary computer for providing perception information to a backup computer plausibility module. The backup computer plausibility monitor can run redundant diverse software on hardware components to detect faults in perception and dynamic driving tasks. The output from the ADAS system 1138 can be provided to a supervisory MCU. If the outputs from the primary and secondary computers conflict, the supervisory MCU must determine how to reconcile the conflict to ensure safe operation.

[0183] In some examples, the host computer can be configured to provide a confidence score to the supervisory MCU indicating the host computer's confidence in the selected result. If the confidence score exceeds a threshold, then the supervisory MCU can follow the host computer's direction, regardless of whether the secondary computer provides conflicting or inconsistent results. In the event that the confidence score does not satisfy the threshold and in the event that the host computer and the secondary computer indicate different results (e.g., a conflict), the supervisory MCU can arbitrate between the computers to determine the appropriate result.

[0184] The supervisory MCU can be configured to run a neural network that is trained and configured to determine conditions under which the secondary computer provides false alarms based at least in part on the output from the host computer and the secondary computer. Thus, the neural network in the supervisory MCU can learn when the output of the secondary computer can be trusted and when it cannot. For example, when the secondary computer is a RADAR-based FCW system, the neural network in the supervisory MCU can learn when the FCW system is identifying metal objects that are not in fact dangerous, such as drain grates or manhole covers that trigger false alarms. Similarly, when the secondary computer is a camera-based LDW system, the neural network in the supervisory MCU can learn to disregard the LDW when a cyclist or pedestrian is present and lane departure is in fact the safest strategy. In embodiments that include a neural network running on the supervisory MCU, the supervisory MCU can include at least one of a DLA or a GPU suitable for running a neural network with associated memory. In preferred embodiments, the supervisory MCU can include and / or be included as a component of the SoC 1104.

[0185] In other examples, the ADAS system 1138 can include a secondary computer that performs ADAS functions using traditional computer vision rules. As such, the secondary computer can use classic computer vision rules (if-then), and the presence of a neural network in the supervisory MCU can improve reliability, safety, and performance. For example, the diverse implementation and intentional non-identity make the overall system more fault-tolerant, especially with respect to faults caused by software (or software-hardware interface) functions. For example, if there is a software bug or error in the software running on the host computer and the non-identical software code running on the secondary computer provides the same overall result, then the supervisory MCU can be more confident that the overall result is correct and that the bug in the software or hardware on the host computer did not cause a substantial error.

[0186] In some examples, the output of the ADAS system 1138 can be fed to a perception block of the host computer and / or a dynamic driving task block of the host computer. For example, if the ADAS system 1138 indicates a forward collision warning due to an object immediately ahead, the perception block can use this information in identifying the object. In other examples, the secondary computer can have its own neural network that is trained and thus reduces the risk of false positives as described herein.

[0187] The vehicle 1100 can also include an infotainment SoC 1130 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as a SoC, the infotainment system can not be a SoC and can include two or more discrete components. The infotainment SoC 1130 can include a combination of hardware and software that can be used to provide audio (e.g., music, a personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., TV, movies, streaming media, etc.), telephony (e.g., hands-free calling), network connectivity (e.g., LTE, WiFi, etc.), and / or information services (e.g., a navigation system, a park assist, a radio data system, vehicle-related information such as a fuel level, a total distance covered, a brake fuel level, an oil level, a door open / close, air filter information, etc.) to the vehicle 1100. For example, the infotainment SoC 1130 can include a radio, a disc player, a navigation system, a video player, USB and Bluetooth connectivity, an in-car computer, in-car entertainment, WiFi, steering wheel audio controls, hands-free voice controls, a heads-up display (HUD), an HMI display 1134, a telematics device, a control panel (e.g., for controlling and / or interacting with various components, features, and / or systems), and / or other components. The infotainment SoC 1130 can further be used to provide information (e.g., visual and / or audible) to a user of the vehicle, such as information from the ADAS system 1138, autonomous driving information such as planned vehicle maneuvers, trajectories, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0188] The infotainment SoC 1130 can include GPU functionality. The infotainment SoC 1130 can communicate with other devices, systems, and / or components of the vehicle 1100 over a bus 1102 (e.g., a CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 1130 can be coupled to a supervisory MCU such that, in the event of a failure of the host controller 1136 (e.g., a primary and / or backup computer of the vehicle 1100), the GPU of the infotainment system can perform some autonomous driving functions. In such examples, the infotainment SoC 1130 can place the vehicle 1100 in a driver safe park mode as described herein.

[0189] The vehicle 1100 can also include an instrument cluster 1132 (e.g., a digital dashboard, electronic instrument cluster, digital instrument panel, etc.). The instrument cluster 1132 can include a controller and / or supercomputer (e.g., a discrete controller or supercomputer). The instrument cluster 1132 can include a set of instruments, such as a speedometer, fuel level, oil pressure, tachometer, odometer, turn indicator, shift position indicator, seat belt warning light, parking brake warning light, engine malfunction light, supplemental restraint system (SRS) system information, lighting controls, safety system controls, navigation information, and so forth. In some examples, information can be displayed and / or shared between the infotainment SoC 1130 and the instrument cluster 1132. In other words, the instrument cluster 1132 can be included as part of the infotainment SoC 1130, or vice versa.

[0190] Figure 11D FIG. 11 illustrates a system diagram of communication between a cloud-based server and an example autonomous vehicle 1100 in accordance with some embodiments of the present disclosure. Figure 11A FIG. 11 illustrates a system diagram of communication between a cloud-based server and an example autonomous vehicle 1100 in accordance with some embodiments of the present disclosure. The system 1176 can include servers 1178, a network 1190, and vehicles including the vehicle 1100. The servers 1178 can include a plurality of GPUs 1184(A)- 1184(H) (collectively referred to herein as GPUs 1184), PCIe switches 1182(A)- 1182(H) (collectively referred to herein as PCIe switches 1182), and / or CPUs 1180(A)- 1180(B) (collectively referred to herein as CPUs 1180). The GPUs 1184, CPUs 1180, and PCIe switches can be interconnected with high-speed interconnects such as, for example and without limitation, NVLink interfaces 1188 developed by NVIDIA and / or PCIe connections 1186. In some examples, the GPUs 1184 are connected via NVLink and / or NVSwitch SoC connections, and the GPUs 1184 and PCIe switches 1182 are connected via PCIe interconnects. Although eight GPUs 1184, two CPUs 1180, and two PCIe switches are illustrated, this is not intended to be limiting. Depending on the embodiment, each of the servers 1178 can include any number of GPUs 1184, CPUs 1180, and / or PCIe switches. For example, each of the servers 1178 can include eight, sixteen, thirty-two, and / or more GPUs 1184.

[0191] The server 1178 can receive image data from vehicles over the network 1190 and representing images showing unexpected or changing road conditions such as a road work that has recently started. The server 1178 can transmit neural networks 1192, updated neural networks 1192, and / or map information 1194, including information about traffic and road conditions, to vehicles over the network 1190. Updates to the map information 1194 can include updates to the HD map 1122, e.g., information about construction sites, potholes, curves, flooding, or other obstacles. In some examples, the neural networks 1192, updated neural networks 1192, and / or map information 1194 can have been produced from experience using training performed at a data center (e.g., using the server 1178 and / or other servers) and / or from data received from any number of vehicles in the environment.

[0192] The server 1178 can be used to train machine learning models (e.g., neural networks) based on training data. The training data can be generated by vehicles and / or can be generated in simulations (e.g., using game engines). In some examples, the training data is labeled (e.g., in cases where the neural network benefits from supervised learning) and / or undergoes other pre-processing, while in other examples, the training data is not labeled and / or pre-processed (e.g., in cases where the neural network does not require supervised learning). The training can be performed according to any one or more categories of machine learning techniques, including but not limited to categories such as: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, federated learning, transfer learning, feature learning (including principal component and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including spare dictionary learning), rule-based machine learning, anomaly detection, and any variants or combinations thereof. Once the machine learning models are trained, the machine learning models can be used by vehicles (e.g., transmitted to vehicles over the network 1190) and / or the machine learning models can be used by the server 1178 to remotely monitor vehicles.

[0193] In some examples, the server 1178 can receive data from vehicles and apply the data to the latest real-time neural networks for real-time intelligent inference. The server 1178 can include deep learning supercomputers and / or specialized AI computers powered by GPUs 1184, such as the DGX and DGX Station machines developed by NVIDIA. However, in some examples, the server 1178 can include deep learning infrastructure of a data center that is powered by CPUs only.

[0194] The deep learning infrastructure of the server 1178 can be capable of fast real-time inference, and can use this capability to assess and validate the health of the processors, software, and / or associated hardware in the vehicle 1100. For example, the deep learning infrastructure can receive periodic updates from the vehicle 1100, such as a sequence of images and / or objects located in the sequence of images that the vehicle 1100 has located (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure can run its own neural network to identify the objects and compare them to the objects identified by the vehicle 1100, and if the results do not match and the infrastructure concludes that the AI in the vehicle 1100 is malfunctioning, the server 1178 can transmit a signal to the vehicle 1100 instructing the fail-safe computer of the vehicle 1100 to take control, notify the passengers, and complete a safe parking operation.

[0195] For inference, the server 1178 can include GPUs 1184 and one or more programmable inference accelerators (such as NVIDIA’s TensorRT). The combination of GPU-powered servers and inference-accelerated can make real-time response possible. In other examples, such as where performance is less important, CPU-, FPGA-, and other processor-powered servers can be used for inference.

[0196] Example Computing Device

[0197] Figure 12 is a block diagram of an example computing device 1200 suitable for implementing some embodiments of the present disclosure. The computing device 1200 can include an interconnection system 1202 that directly or indirectly couples the following devices: memory 1204, one or more central processing units (CPUs) 1206, one or more graphics processing units (GPUs) 1208, a communication interface 1210, input / output (I / O) ports 1212, I / O components 1214, a power supply 1216, one or more presentation components 1218 (e.g., display(s)), and one or more logic units 1220. In at least one embodiment, the computing device(s) 1200 can include one or more virtual machines (VMs), and / or any component thereof can include a virtual component (e.g., a virtual hardware component). For non-limiting examples, one or more of the GPUs 1208 can include one or more vGPUs, one or more of the CPUs 1206 can include one or more vCPUs, and / or one or more of the logic units 1220 can include one or more virtual logic units. As such, the computing device(s) 1200 can include discrete components (e.g., a full GPU dedicated to the computing device 1200), virtual components (e.g., a portion of a GPU dedicated to the computing device 1200), or a combination thereof.

[0198] Although Figure 12 various blocks of the computing device are shown as being connected by the interconnect system 1202, this is not intended to be limiting and is for clarity only. For example, in some embodiments, a presentation component 1218 (such as a display device) can be considered an I / O component 1214 (e.g., if the display is a touch screen). As another example, the CPU 1206 and / or GPU 1208 can include memory (e.g., the memory 1204 can represent a storage device in addition to the memory of the GPU 1208, CPU 1206, and / or other components). In other words, Figure 12 the computing device of FIG. 12 is illustrative. No distinction is made between Figure 12 such categories like “workstation,” “server,” “laptop,” “desktop,” “tablet,” “client device,” “mobile device,” “handheld device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types, as all are considered within the scope of the computing device of FIG. 12.

[0199] The interconnect system 1202 can represent one or more links or buses, such as an address bus, data bus, control bus, or a combination thereof. The interconnect system 1202 can include one or more bus or link types, such as an Industry Standard Architecture (ISA) bus, an Extended Industry Standard Architecture (EISA) bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe) bus, and / or another type of bus or link. In some embodiments, there are direct connections between components. As an example, the CPU 1206 can be directly connected to the memory 1204. Further, the CPU 1206 can be directly connected to the GPU 1208. Where there are direct or point-to-point connections between components, the interconnect system 1202 can include a PCIe link to perform the connection. In these examples, a PCI bus need not be included in the computing device 1200.

[0200] The memory 1204 can include any of a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by the computing device 1200. Computer-readable media can include both volatile and nonvolatile media, and removable and non-removable media. By way of example, and not limitation, computer-readable media can comprise computer storage media and communication media.

[0201] Computer storage media can include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules, and / or other data types. For example, memory 1204 can store computer readable instructions (e.g., representing program(s) and / or program element(s), such as an operating system). Computer storage media can include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by computing device 1200. As used herein, computer storage media does not include a signal per se.

[0202] Computer storage media can embody computer readable instructions, data structures, program modules, and / or other data types in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. The term “modulated data signal” can refer to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, computer storage media can include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of the any of the above should also be included within the scope of computer readable media.

[0203] CPUs 1206 can each include one or more cores capable of handling a multitude of software threads concurrently (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.). CPUs 1206 can include any type of processors, and can include different types of processors depending on the type of computing device 1200 implemented (e.g., less powerful processors for mobile devices and more powerful processors for servers). For example, depending on the type of computing device 1200, the processors can be Advanced RISC Machines (ARM) processors implemented using Reduced Instruction Set Computing (RISC) or x86 processors implemented using Complex Instruction Set Computing (CISC). Computing device 1200 can also include one or more CPUs 1206 in addition to, or in place of, one or more microprocessors or co-processors such as mathematical co-processors.

[0204] In addition to, or in place of, CPU(s) 1206, GPU(s) 1208 can be configured to execute at least some of the computer-readable instructions to control one or more components of computing device 1200 to perform one or more of the methods and / or processes described herein. One or more of GPUs 1208 can be integrated GPUs (e.g., with one or more of CPUs 1206) and / or one or more of GPUs 1208 can be discrete GPUs. In embodiments, one or more of GPUs 1208 can be a co-processor of one or more of CPUs 1206. GPUs 1208 can be used by computing device 1200 to render graphics (e.g., 3D graphics) or to perform general purpose computing. For example, GPUs 1208 can be used for general purpose computing on GPUs (GPGPU). GPUs 1208 can include hundreds or thousands of cores capable of handling hundreds or thousands of software threads concurrently. GPUs 1208 can generate pixel data for output images in response to rendering commands (e.g., received from CPUs 1206 via a host interface). GPUs 1208 can include graphics memory (e.g., display memory) for storing pixel data or any other suitable data (e.g., GPGPU data). The display memory can be included as part of memory 1204. GPUs 1208 can include two or more GPUs operating in parallel (e.g., via a link). The link can connect the GPUs directly (e.g., using NVLINK) or can connect the GPUs through a switch (e.g., using an NVSwitch). When combined together, each GPU 1208 can generate pixel data or GPGPU data for a different portion of an output or for a different output (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU can include its own memory, or can share memory with other GPUs.

[0205] In addition to or in lieu of CPU(s) 1206 and / or GPU(s) 1208, logic unit(s) 1220 can be configured to execute at least some of the computer-readable instructions to control one or more components of computing device 1200 to perform one or more of the methods and / or processes described herein. In embodiments, CPU(s) 1206, GPU(s) 1208, and / or logic unit(s) 1220 can perform any combination of the methods, processes, and / or portions thereof, discretely or jointly. One or more of logic unit(s) 1220 can be part of one or more of CPU(s) 1206 and / or integrated in one or more of CPU(s) 1206 and / or one or more of logic unit(s) 1220 can be a discrete component or otherwise external to CPU(s) 1206 and / or GPU(s) 1208. In embodiments, one or more of logic unit(s) 1220 can be a co-processor of one or more of CPU(s) 1206 and / or one or more of GPU(s) 1208.

[0206] Examples of logic unit(s) 1220 include one or more processing cores and / or components thereof, such as a data processing unit (DPU), a tensor core (TC), a tensor processing unit (TPU), a pixel visual core (PVC), a visual processing unit (VPU), a graphics processing cluster (GPC), a texture processing cluster (TPC), a streaming multi-processor (SM), a tree traversal unit (TTU), an artificial intelligence accelerator (AIA), a deep learning accelerator (DLA), an arithmetic logic unit (ALU), an application-specific integrated circuit (ASIC), a floating-point unit (FPU), an input / output (I / O) element, a peripheral component interconnect (PCI) or peripheral component interconnect express (PCIe) element, etc.

[0207] The communication interface 1210 can include one or more receivers, transmitters and / or transceivers that enable the computing device 1200 to communicate with other computing devices via electronic communication networks, including wired and / or wireless communications. The communication interface 1210 can include components and functionality to enable communication over any of a plurality of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., through Ethernet or InfiniBand), low power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, the logic unit 1220 and / or the communication interface 1210 can include one or more data processing units (DPUs) to transfer data received over a network and / or through the interconnect system 1202 directly to one or more GPUs 1208 (e.g., memory thereof).

[0208] The I / O ports 1212 can enable the computing device 1200 to be logically coupled to other devices including the I / O components 1214, the presentation components (1218), and / or other components, some of which can be built into (e.g., integrated in) the computing device 1200. Illustrative I / O components 1214 include a microphone, mouse, keyboard, joystick, game pad, game controller, satellite dish, scanner, printer, wireless device, etc. The I / O components 1214 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by a user. In some instances, inputs can be transmitted to appropriate network elements for further processing. A NUI can implement any combination of speech recognition, pen / mouse-based recognition, facial recognition, biometric recognition, gesture recognition (both on- screen and adjacent to the screen), air gestures, head and eye tracking, and touch recognition (as described in more detail below) associated with a display of the computing device 1200. The computing device 1200 can include a depth camera, such as a stereoscopic camera system, an infrared camera system, an RGB camera system, touch screen technology, and combinations of these, for gesture detection and recognition. In addition, the computing device 1200 can include an accelerometer or a gyroscope (e.g., as part of an inertial measurement unit (IMU)) to detect motion. In some examples, the computing device 1200 can use the output of the accelerometer or gyroscope to render an immersive augmented reality or virtual reality.

[0209] The power supply 1216 can include a hard-wired power supply, a battery power supply, or a combination thereof. The power supply 1216 can provide power to the computing device 1200 to enable the components of the computing device 1200 to operate.

[0210] The presentation component 1218 can include a display (e.g., a monitor, a touchscreen, a television screen, a heads-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. The presentation component 1218 can receive data from other components (e.g., the GPU 1208, the CPU 1206, the DPU, etc.) and output the data (e.g., as images, video, sound, etc.).

[0211] Example data center

[0212] Figure 13 An example data center 1300 that can be used in at least one embodiment of the present disclosure is shown. The data center 1300 can include a data center infrastructure layer 1310, a framework layer 1320, a software layer 1330, and / or an application layer 1340.

[0213] As Figure 13 shown, the data center infrastructure layer 1310 can include a resource orchestrator 1312, grouped computing resources 1314, and node computing resources (“node C.R.s”) 1316(1)-1316(N), where “N” represents any whole, positive integer. In at least one embodiment, the node C.R.s 1316(1)-1316(N) can include, but are not limited to, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic random access memory), storage devices (e.g., solid state or disk drives), network input / output (NW I / O) devices, network switches, virtual machines (VMs), power modules, and / or cooling modules, etc. In some embodiments, one or more of the node C.R.s 1316(1)-1316(N) can correspond to a server having one or more of the above-described computing resources. Further, in some embodiments, the node C.R.s 1316(1)-1316(N) can include one or more virtual components, such as vGPUs, vCPUs, etc., and / or one or more of the node C.R.s 1316(1)-1316(N) can correspond to a virtual machine (VM).

[0214] In at least one embodiment, the grouped computing resources 1314 may include individual groups of node CRs 1316 housed within one or more racks (not shown), or a plurality of racks housed within a data center in different geographical locations (also not shown). Individual groups of node CRs 1316 within the grouped computing resources 1314 may include grouped computing, networking, memory, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, a plurality of node CRs 1316, including CPUs, GPUs, DPUs, and / or other processors, may be grouped within one or more racks to provide computing resources to support one or more workloads. One or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0215] Resource coordinator 1312 may be configured or otherwise control one or more nodes CRs 1316(1)-1316(N) and / or grouped computing resources 1314. In at least one embodiment, resource coordinator 1312 may include a Software Design Infrastructure (SDI) management entity for data center 1300. Resource coordinator 1312 may include hardware, software, or some combination thereof.

[0216] In at least one embodiment, such as Figure 13 As shown, framework layer 1320 may include a job scheduler 1333, a configuration manager 1334, a resource manager 1336, and / or a distributed file system 1338. Framework layer 1320 may include a framework for software 1332 supporting software layer 1330 and / or one or more applications 1342 of application layer 1340. Software 1332 or application 1342 may respectively contain web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. Framework layer 1320 may be, but is not limited to, free and open-source software web application frameworks (such as Apache Spark) that can utilize distributed file system 1338 for large-scale data processing (e.g., big data). TM(“Spark” hereinafter). In at least one embodiment, job scheduler 1333 can include a Spark driver to facilitate scheduling workloads supported by different tiers of data center 1300. Configuration manager 1334 can be capable of configuring different tiers, such as software tier 1330 and framework tier 1320 (which includes Spark and distributed file system 1338 for supporting large scale data processing). Resource manager 1336 can be capable of managing clustered or grouped computing resources mapped to or allocated for supporting distributed file system 1338 and job scheduler 1333. In at least one embodiment, clustered or grouped computing resources can include grouped computing resources 1314 of data center infrastructure layer 1310. Resource manager 1336 can coordinate with resource orchestrator 1312 to manage these mapped or allocated computing resources.

[0217] In at least one embodiment, software 1332 included in software tier 1330 can include software used by at least portions of node C.R.s 1316(1)-1316(N), grouped computing resources 1314, and / or distributed file system 1338 of framework tier 1320. One or more types of software can include, but are not limited to, internet web page search software, email virus scanning software, database software, and streaming video content software.

[0218] In at least one embodiment, applications 1342 included in application tier 1340 can include one or more types of applications used by at least portions of node C.R.s 1316(1)-1316(N), grouped computing resources 1314, and / or distributed file system 1338 of framework tier 1320. One or more types of applications can include, but are not limited to, any number of genomics applications, cognitive computing and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments.

[0219] In at least one embodiment, any of configuration manager 1334, resource manager 1336, and resource orchestrator 1312 can implement any number and type of self-modifying actions based on any quantity and type of data acquired in any technically feasible manner. Self-modifying actions can free data center operators of data center 1300 from making possibly poor configuration decisions and possibly avoid underutilization and / or poor performance portions of a data center.

[0220] According to one or more embodiments described herein, data center 1300 may include tools, services, software, or other resources to train one or more machine learning models or to use one or more machine learning models to predict or infer information. For example, one or more machine learning models may be trained by using the software and / or computing resources described above with respect to data center 1300 to compute weight parameters according to a neural network architecture. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks may be used to infer or predict information using the resources described above with respect to data center 1300 by using weight parameters computed through one or more training techniques (such as, but not limited to, those described herein).

[0221] In at least one embodiment, the data center 1300 may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, and / or other hardware (or corresponding virtual computing resources) to perform training and / or inference using the aforementioned resources. Furthermore, one or more of the software and / or hardware resources described above may be configured to allow a user to train or perform services that infer information, such as image recognition, speech recognition, or other artificial intelligence services.

[0222] Example network environment

[0223] A network environment suitable for implementing embodiments of this disclosure may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. Client devices, servers, and / or other device types (e.g., each device) may be... Figure 12 This is implemented on one or more instances of computing device 1200—for example, each device may include similar components, features, and / or functions of computing device 1200. Furthermore, in the case of implementing backend devices (e.g., servers, NAS, etc.), the backend devices may be included as part of data center 1300, examples of which are described herein. Figure 13 To describe in more detail.

[0224] Components of a network environment can communicate with each other via a network, which can be wired, wireless, or both. A network can include multiple networks or one of multiple networks. For example, a network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. Where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) can provide wireless connectivity.

[0225] Compatible network environments can include one or more peer-to-peer network environments (in which case servers can not be included in the network environment) and one or more client-server network environments (in which case one or more servers can be included in the network environment). In a peer-to-peer network environment, functionality described herein for servers can be implemented on any number of client devices.

[0226] In at least one embodiment, the network environment can include one or more cloud-based network environments, distributed computing environments, combinations thereof, and the like. A cloud-based network environment can include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which can include one or more core network servers and / or edge servers. The framework layer can include a framework that supports one or more applications of a software layer and / or an application layer. The software or applications can include network-based service software or applications, respectively. In embodiments, one or more client devices can use the network-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer can be, without limitation, a free and open-source software web application framework as can be used for large-scale data processing (e.g., “big data”) using a distributed file system.

[0227] The cloud-based network environment can provide cloud computing and / or cloud storage that performs any combination of the computing and / or data storage functionality described herein (or one or more portions thereof). Any of these different functionalities can be distributed across multiple locations from central or core servers (e.g., one or more data centers that can be distributed across a state, a region, a country, globally, etc.). The core servers can designate at least a portion of the functionality to edge servers if the connection to the user (e.g., client device) is relatively close to the edge servers. The cloud-based network environment can be private (e.g., limited to a single organization), can be public (e.g., available to many organizations), and / or combinations thereof (e.g., a hybrid cloud environment).

[0228] The client device(s) can include the client devices described herein with respect to Figure 12At least some of the components, features and functionality of the described example computing device 1200 are described. By way of example, and not limitation, a client device can be implemented as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, video camera, surveillance device or system, vehicle, boat, spaceship, virtual machine, drone, robot, handheld communication device, hospital device, gaming device or system, entertainment system, vehicle computer system, embedded system controller, remote control, appliance, consumer electronic device, workstation, edge device, any combination of these depicted devices, or any other suitable device.

[0229] The present disclosure can be described in the general context of machine-usable instructions or computer code, including computer-executable instructions such as program modules, being executed by a computer or other machine, such as a personal data assistant or other handheld device. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. The present disclosure can be practiced in a variety of system configurations, including hand-held devices, consumer electronics, general- purpose computers, more specialty computing devices, etc. The present disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network.

[0230] As used herein, the term “and / or” with respect to two or more elements should be interpreted as only one element or a combination of elements. For example, “element A, element B, and / or element C” can include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or element A, B, and C. In addition, “at least one of element A or element B” can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Further, “at least one of element A and element B” can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0231] The subject matter of the present disclosure is described with specificity herein to meet statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors have contemplated that the claimed subject matter might also be embodied in other ways, to include different steps or combinations of steps similar to the ones described in this document, in conjunction with other present or future technologies. Moreover, although the terms "step" and / or "block" might be used herein to connote different elements of methods employed, the terms should not be interpreted as implying any particular order among or between various steps herein disclosed unless and except when the order of individual steps is explicitly described.

Claims

1. A method for tracking an object, comprising: determining, based at least on first image data representing a first image, a first enclosing shape associated with an object depicted in the first image and a first feature point associated with the object that is located within the first enclosing shape; determining, based at least on second image data representing a second image and the first enclosing shape, a second enclosing shape associated with the second image and a second feature point that is possibly associated with the object that is located within the second enclosing shape; determining, based at least on a comparison of the first feature point associated with the object and the second feature point that is possibly associated with the object, a scalar change associated with the object; determining, based at least on the scalar change, that the object is depicted in the second image; performing, based at least on the object being depicted in the second image, one or more operations on a track corresponding to the object; and causing a machine to navigate based at least on the tracking corresponding to the object.

2. The method of claim 1, wherein determining the second feature point that is possibly associated with the object comprises: determining the second enclosing shape by at least reducing a size of the first enclosing shape from the first image; determining a location of the second enclosing shape in the second image based at least on a location of the first enclosing shape in the first image; and determining that the second feature point is located within the second enclosing shape in the second image.

3. The method of claim 1, further comprising: determining, based at least on the scalar change, at least one of the first feature point and at least one of the second feature point, a translation associated with the object; wherein determining that the object is depicted in the second image is further based at least on the translation.

4. The method of claim 1, wherein determining that the object is depicted in the second image comprises: determining, based at least on the first enclosing shape and the scalar change, a third enclosing shape; determining, based at least on the second image data, a fourth enclosing shape associated with the object; and determining, based at least on a comparison of the third enclosing shape and the fourth enclosing shape, that the object is depicted in the second image.

5. The method of claim 4, wherein determining that the object is depicted in the second image comprises: determining an amount of overlap between the fourth enclosing shape and the third enclosing shape; determining, based at least on the amount of overlap, a score associated with the object; and determining, based at least on the score, that the object is depicted in the second image.

6. The method of claim 1, wherein determining the scalar change comprises: determining, from the first feature point, a first distance between the first feature point and the second feature point; determining, from the second feature point, a second distance between a third feature point and a fourth feature point, wherein the third feature point is associated with the first feature point and the fourth feature point is associated with the second feature point; and determining, based at least on the first distance and the second distance, the scalar change. ​ ​ ​ ​ ​ 7. The method of claim 1, further comprising: determining, based at least on the object being depicted in the second image, a confidence score associated with the object; wherein performing the one or more operations comprises generating the trajectory associated with the object based at least on the confidence score.

8. The method of claim 7, further comprising: determining, based at least on third image data representing a third image, a third feature point associated with the third image; determining, based at least on the second feature point and the third feature point, an additional scalar change associated with the object; determining, based at least on the additional scalar change, that the object is not depicted in the third image; and one of: determining, based at least on the object being depicted in the third image, an additional confidence score associated with the object by increasing the confidence score; or determining, based at least on the object not being depicted in the third image, the additional confidence score by decreasing the confidence score.

9. The method of claim 1, wherein the one or more operations comprise: generating the trajectory associated with the object, and wherein the method further comprises: determining, based at least on third image data representing a third image, a third feature point associated with the third image; determining, based at least on the second feature point and the third feature point, an additional scalar change associated with the object; determining, based at least on the additional scalar change, that the object is not depicted in the third image; and terminating the trajectory associated with the object based at least on the object not being depicted in the third image.

10. A system for determining a time-to-collision (TTC) estimate associated with an object, comprising: one or more processors to: determine, based at least on first image data representing a first image, a first feature point located on an object depicted in the first image; determine, based at least on second image data representing a second image, a second feature point located on the object depicted in the second image; determine, based at least on the first feature point and the second feature point, a scalar change in the object between the first image and the second image; and determine, based at least on the scalar change, a time-to-collision (TTC) value associated with the object.

11. The system of claim 10, wherein the second feature point is determined based at least on: determining, based at least on the first image data, a first enclosing shape associated with the object; determining, by adjusting at least a size of the first enclosing shape, a second enclosing shape associated with the object; and determining, based at least on the second image data, that the second feature point is located within the second enclosing shape.

12. The system of claim 10, wherein determining the scalar change is based at least on: determining, from the first feature point, a first distance between the first feature point and the second feature point; determining, from the second feature point, a second distance between a third feature point and a fourth feature point, wherein the third feature point is associated with the first feature point and the fourth feature point is associated with the second feature point; and ​ ​ determine the scalar change based at least on the first distance and the second distance.

13. The system of claim 10, wherein determining the scalar change is based at least on: determining a first distance between a pair of first feature points from the first feature points; determining a second distance between a pair of second feature points from the second feature points, wherein the pair of second feature points is associated with the pair of first feature points; determining the scalar change based at least on the first distance and the second distance; and determining the scalar change as at least one of a median of the scalar changes or a mean of the scalar changes.

14. The system of claim 10, wherein the one or more processors are further to: determine a time difference between the first image and the second image; wherein the TTC is further determined based at least on the time difference.

15. The system of claim 10, wherein the one or more processors are further to: determine, based at least on third image data representative of a third image, a third feature point associated with the object depicted in the third image; determine, based at least on the first feature point and the third feature point, an additional scalar change associated with the object; determine, based at least on the additional scalar change, an additional TTC associated with the object; and determine, based at least on the TTC and the additional TTC, a final TTC associated with the object.

16. The system of claim 10, wherein the one or more processors are further to: determine, based at least on the first image data, a first enclosing shape associated with the object; determine, based at least on the first enclosing shape and the scalar change, a second enclosing shape associated with the object; determine, based at least on the second image data, a third enclosing shape associated with the object; and determine, based at least on the second enclosing shape and the third enclosing shape, that the scalar change is validated.

17. The system of claim 10, wherein the one or more processors are further to: determine, based at least on the scalar change, at least one of the first feature point, and at least one of the second feature point, a translation associated with the object; and determine, based at least on the scalar change and the translation, that the object depicted in the first image and the object depicted in the second image are the same object.

18. The system of claim 10, wherein the system is included in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing optical transport simulation; a system for generating or presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system implemented using an edge device; a system implemented using a robot; ​ A system for performing conversational AI operations; A system for generating synthetic data; A system that consolidates one or more virtual machines (VMs); A system implemented at least in part in a data center; or A system implemented at least in part using cloud computing resources.

19. A processor comprising: processing circuitry to perform one or more operations based at least on a scalar change of an object between a first image and a second image, wherein the scalar change is determined based at least on a comparison between a first feature point at a first location of the object as depicted in the first image and a second feature point at a second location of the object as depicted in the second image.

20. The processor of claim 19, wherein the one or more operations include at least one of: determining that the object is depicted in the second image based at least on the scalar change; or determining a time to collision (TTC) associated with the object based at least on the scalar change.

21. The processor of claim 19, wherein the processor is included in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing optical transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system implemented using edge devices; a system for generating or presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system implemented using robots; a system for performing conversational AI operations; a system for generating synthetic data; a system that consolidates one or more virtual machines (VMs); a system implemented at least in part in a data center; or a system implemented at least in part using cloud computing resources.

Images