Object tracking using LiDAR data for autonomous machine applications

Abstract

To provide object tracking that uses LiDAR data for autonomous machine applications.SOLUTION: In various examples, an obstacle detector has a capability to track a velocity state of detected objects or obstacles, using LiDAR data. For example, by using the LiDAR data alone, an iterative closest point (ICP) algorithm may be used to determine a current state of detected objects for a current frame and a Kalman filter may be used to maintain a tracked state of one or a plurality of objects detected over time. The obstacle detector may be configured to estimate a velocity for the one or the plurality of detected objects; compare the estimated velocity with tracked states of one or a plurality of previous tracked states of previously detected objects; determine that the detected object corresponds to a certain previously detected object; and update the tracked state of the previously detected object with the estimated velocity.SELECTED DRAWING: Figure 1A

Description

Background Art

[0001] LiDAR (light detection and ranging) can be used to detect objects with high precision. For example, a LiDAR sensor measures the return time of a laser (or other source) from an object, and the accumulation of these measurements can be used to generate a three-dimensional (3D) point cloud that represents the objects within the LiDAR sensor's perception field or field of view. As such, each sweep of the LiDAR sensor can be used to generate a 3D point cloud that is a single capture at the time of reading from the LiDAR sensor, and subsequent LiDAR sweeps can be separated by time intervals. In one exemplary field of use, LiDAR sensors are used in autonomous or semi-autonomous vehicles and other machine types to detect potential objects in proximity to the LiDAR sensor and, by extension, in proximity to the vehicle or machine. For example, a LiDAR sensor can be used to identify the boundary shape corresponding to an object in the environment with a high level of accuracy. However, many types of LiDAR sensors do not capture velocity information, so the boundary shapes generated from LiDAR data typically do not have velocity information. Without velocity information, these boundary shapes have conventionally not been used for vehicle control. For example, when the ego machine is moving forward, if an object appears in front of the ego machine at a certain distance, different actions may be required compared to the case where the object is moving away from the ego machine (e.g., at the same speed as or faster than the ego machine), especially when the object is moving towards the ego machine. Therefore, due to this lack of velocity information, alternative sources of velocity information have conventionally been required to gather sufficient information about the movement of objects for making accurate and reliable control decisions.

Prior Art Documents

Patent Documents

[0002]

Patent Document 1

[0003] Embodiments of this disclosure relate to object tracking, at least partially via LiDAR data, for autonomous machine applications, such as autonomous or semi-autonomous vehicles or machine and / or robotic platforms. Systems and methods are disclosed for identifying and tracking objects by estimating velocity and comparing the estimated velocity with one or more tracked object states. Based on the comparison, a particular tracked object state is identified as corresponding to a particular object and updated with new estimated velocity data.

[0004] In contrast to these conventional methods, current systems and methods can use LiDAR data to track the velocity state of detected objects or obstacles. For example, using LiDAR data alone, an iterative nearest nearest (ICP) algorithm may be used to determine the current state of a detected object in the current frame, and a Kalman filter may be used to maintain the tracking state of one or more detected objects over time. Embodiments of the present disclosure include a system configured to estimate the velocity of one or more detected objects, compare the estimated velocity to the previous tracking state of one or more previously detected objects, determine that a detected object corresponds to a particular previously detected object, and update the tracking state of the previously detected object with the estimated velocity. In such a configuration, between each LiDAR sweep, the system can associate each detected object with a detected object from the previous LiDAR sweep and update the known information about those detected objects. The tracking status of each detected object can be represented, for example, using a Kalman filter to generate a probability distribution function (PDF), where the PDF represents the probability of the detected object's velocity (and / or its position in the environment, as determined by the velocity). As new data corresponding to subsequent frames is analyzed by the system, the tracking status can be refined over time.

[0005] The system and method for object tracking using LiDAR data for autonomous machine applications are described in detail below with reference to the attached drawings. [Brief explanation of the drawing]

[0006] [Figure 1A] This is a data flow diagram illustrating an object tracking system according to some embodiments of the present disclosure. [Figure 1B] This is a time flowchart illustrating a process for object tracking according to some embodiments of the present disclosure. [Figure 2A]A graphic representation of a first set of LiDAR data at time T1, showing an object to be tracked, according to some embodiments of the present disclosure. [Figure 2B] A graphical representation of a second set of LiDAR data at time T2, after T1, showing the updated location of an object to be tracked, according to some embodiments of the present disclosure. [Figure 3] This is a graphical representation of the probability distribution corresponding to the velocity of a detected object, according to some embodiments of the present disclosure. [Figure 4] Illustrations illustrating how an ego vehicle passes a parked vehicle and how the parked vehicle is detected by the ego vehicle, according to some embodiments of the present disclosure. [Figure 5A] Illustrations of exemplary autonomous vehicles according to some embodiments of the present disclosure. [Figure 5B] Figure 5A shows examples of camera positions and fields of view of an exemplary autonomous vehicle according to some embodiments of the present disclosure. [Figure 5C] Figure 5A is a block diagram of an exemplary system architecture of an exemplary autonomous vehicle according to some embodiments of the present disclosure. [Figure 5D] This is a system diagram of communication between a cloud-based server and the exemplary autonomous vehicle shown in Figure 5A, according to some embodiments of the present disclosure. [Figure 6] This is a block diagram of an exemplary computing device suitable for use in implementing some embodiments of the present disclosure. [Figure 7] This is an exemplary data center block diagram suitable for use in implementing some embodiments of the present disclosure. [Modes for carrying out the invention]

[0007] This disclosure relates to systems and methods for object tracking via the use of LiDAR data. This disclosure may be described in relation to an exemplary autonomous vehicle 500 (referred to herein as "vehicle 500" or "ego-machine 500" as otherwise provided herein, examples of which are illustrated with reference to Figures 5A–5D), but this is not intended to be limiting. For example, the systems and methods described herein may be used, without limitation, by non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more adaptive driver assistance systems (ADAS)), piloted or unpiloted robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles towed to one or more trailers, flying ships, boats, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, submarines, drones, and / or other vehicle types. In addition, while this disclosure may describe object tracking via LiDAR data in autonomous vehicles, this is not intended to be an limitation, and the systems and methods described herein may be used in any other technological space where augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications, and / or safety applications may be used.

[0008] Embodiments of the present disclosure relate to object tracking at least partially via LiDAR data in various applications—for example, autonomous or semi-autonomous machine applications. For example, one or more LiDAR sensors may generate 3D LiDAR data representing their field of view or perceptual field—for example, the LiDAR data may represent a standard spherical projection or other projection type. A LiDAR sensor may generate LiDAR data sequentially such that a number of sets of LiDAR data are generated—each representing a single sweep or rotation of the LiDAR sensor. Sets of LiDAR data may be separated by time intervals, and embodiments of the present disclosure may use a first (for example, current) set of LiDAR data and compare it to a tracking state representing one or more previous sets of LiDAR data, as described herein.

[0009] In its initial state, the system can receive LiDAR data (e.g., in the form of a 3D point cloud) generated using one or more LiDAR sensors associated with an autonomous or semi-autonomous vehicle or other machine. Based on the LiDAR data, the system can identify one or more objects—for example, by detecting boundaries within the LiDAR data corresponding to each object. In this initial state, the system may not have information, if any, about which of the objects are moving (and if so, the speed and direction of such movement). To account for this lack of information, the system can initially create a velocity tracking state for an empty object. During subsequent LiDAR data generation, the system can estimate and refine the velocity tracking state of one or more of the detected objects.

[0010] In embodiments of the present disclosure, the system can determine the estimated velocity of an object detected in a subsequent sweep ("current object detection") based at least in part on a comparison of the current object detection, one or more previous object detections detected between previous LiDAR sweeps, and the time interval between each LiDAR sweep. Thus, the current object detection may have multiple estimated velocities based on one or more previous object detections in previous sweeps that the current detection can be compared to. A cross-sectional distance can be calculated between the location of the current object detection and the location of the previous object detection (based on the physical distance between these two locations). Coordinate transformations—for example, the current frame into the current coordinate system—can also be applied so that the cross-sectional distance accounts for the ego motion of the LiDAR sensor between sweeps (e.g., the motion of the ego machine between consecutive LiDAR sweeps). The cross-sectional distance can then be divided by the elapsed time between the current object detection and the previous object detection, based on the time interval between sweeps of the LiDAR sensor, in order to calculate the estimated velocity. The estimated velocity may include a magnitude component and / or a directional component.

[0011] In embodiments of this disclosure, the system can compare each estimated velocity with its corresponding previous object detection. For example, each previous object detection may have a velocity tracking state (or more generally, a tracked object state) indicating the estimated velocity of that previous object detection, if any, which is refined over time. The comparison may also be between a first object boundary from a previous sweep and a second object boundary from the current sweep. The comparison may be performed using an iterative nearest nearest (ICP) algorithm which can provide individual comparison points of LiDAR data corresponding to each object boundary. The system can then use the probability distribution function of objects ("PDF") as determined from the tracking state to determine the likelihood that the estimated velocity corresponds to each previously detected object.

[0012] In embodiments of the present disclosure, the system can determine, at least in part, based on a comparison of each estimated velocity with its corresponding previously detected object, that one of the previously detected objects has the highest probability of corresponding to the current object. First, to reduce computation and execution time, the system can determine a minimum threshold for comparison—for example, using the Mahalanobis distance. For example, if the estimated velocity exceeds a minimum threshold with respect to the PDF of a particular object, the system can exclude that particular object from consideration. For the remaining objects, the estimated velocity can be compared to the PDF using a cost function that represents the probability that the currently detected object corresponds to one or more of the previously detected objects. The system can then evaluate the cost function and determine which object—if any—to associate with the current velocity, and update the tracking status corresponding to the associated object.

[0013] In embodiments of this disclosure, the system can generate an updated tracked object state by updating the tracked object state corresponding to the previous object detection having the highest probability corresponding to the current object detection. For example, the current estimated velocity may be applied to the tracked object state, and a Kalman filter may be used to update the tracked object state using the estimated velocity. This updated tracked object state can then be the basis for comparisons between subsequent LiDAR sweeps so as to continue refining the tracked object state over time through the use of the Kalman filter. Thus, LiDAR data can be used directly to identify velocity information corresponding to objects or obstacles in the environment, and it can be made possible to apply LiDAR data alone directly to control decisions of the ego machine.

[0014] Referring to Figure 1A, Figure 1A is an exemplary object tracking system 100 (or referred to as "System 100") according to some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are described merely as examples. Other configurations and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those illustrated, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as individual or distributed components or in combination with other components, and in any appropriate combination and location. The various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, the various functions may be performed by a processor that executes instructions stored in memory. For example, in some embodiments, System 100 may include similar features, functionalities, and / or components to those of the exemplary autonomous vehicle 500 in Figures 5A–5D, the exemplary computing device 600 in Figure 6, and / or the exemplary data center 700 in Figure 7.

[0015] System 100 may be any of the various systems, a component thereof, or may be related to any of the various systems in other ways. For example, System 100 may include, or correspond to, a control system for an autonomous or semi-autonomous machine, a perception system for an autonomous or semi-autonomous machine, a system for performing simulation operations, a system for performing deep learning operations, a system implemented using edge devices, a system implemented using robots, a system incorporating one or more virtual machines (VMs), a system at least partially implemented in a data center, and / or a system at least partially implemented using cloud computing resources.

[0016] As shown in FIG. 1A, system 100 may include one or more depth perception sensors 102 (e.g., LiDAR sensors, RADAR sensors, ultrasonic sensors, etc.). The depth perception sensors 102 can generally include a transmitter and a receiver, and can include any suitable field of view or perception field - for example, a wide field of view (e.g., from 180 degrees to 360 degrees) - and can move (e.g., rotate) in an embodiment to obtain a larger area of the field of view associated with the depth perception sensors 102. A signal - for example, a LiDAR signal - can be reflected by an object near the depth perception sensors 102. The object may be moving relative to the depth perception sensors 102, and the depth perception sensors 102 may be moving relative to a base surface (e.g., the road on which the autonomous vehicle 500 is driving). The receiver can receive the indications of these various reflected signals (directly or indirectly), and the indications can be stored and / or transmitted as data for later analysis.

[0017] The depth perception sensors 102 may have a sensor controller 104 that can be used to control the operation of the depth perception sensors 102 and interpret the results. For example, the sensor controller 104, or another processor, can receive sensor data 106 and process, analyze, or otherwise execute calculations related to the sensor data 106. In some embodiments, the depth perception sensors 102 and / or the sensor controller 104 may be similar to the LiDAR sensors 564 described with respect to FIGS. 5A - 5C, or may be another type of depth perception sensor 102.

[0018] The sensor controller 104 can output sensor data 106 to a computing system 108, such as a computing system that executes within the vehicle 500 of FIG. 6 and / or an exemplary computing device 600. The sensor data 106 can be in any of a variety of forms, such as, without limitation, (2D or 3D) LiDAR point clouds, projected or range images, and / or other sensor data representations. The sensor data 106 can be analyzed to perform various functions related thereto and can be used in conjunction with other sensor data 106 (such as the various sensors shown in FIGS. 5A - 5C and discussed herein). In embodiments where a LiDAR sensor is used, the sensor data 106 may be referred to as LiDAR data, but in other embodiments of the present disclosure, the sensor data 106 may be another type of depth data (such as from RADAR, ultrasonic, etc.).

[0019] The system 100 can include an object detector 110. The object detector 110 can analyze a set of LiDAR data to identify one or more obstacle boundaries. The obstacle boundaries can be identified from a set of LiDAR data that indicates physical obstacles near the depth perception sensor 102. The obstacle boundaries can indicate a portion of a physical obstacle facing the depth perception sensor 102. In an embodiment, the continuity of the obstacle boundaries and additional obstacle boundaries can be occluded from the depth perception sensor 102. Thus, when the machine 500 and / or an obstacle are moving relative to each other, existing obstacle boundaries can be extended (for example, as shown in FIG. 4 and discussed herein, because additional portions of the physical obstacle are now visible), and new or updated obstacle boundaries can be created (for example, because the corresponding physical obstacle is no longer occluded). Similarly, two or more obstacle boundaries may correspond to the same physical obstacle, for example, due to the irregular shape of the physical obstacle.

[0020] An example of LiDAR data and identified obstacle boundaries is shown in Figures 2A and 2B. A first set of LiDAR data 200 is shown in Figure 2A, and a second set of LiDAR data 202 is shown in Figure 2B. Obstacle boundaries 204, corresponding to detected obstacle boundaries, are shown in Figures 2A and 2B. Figures 2A and 2B also include a central region 206 and its radius extending from it, which corresponds to (but is not shown in Figure 2A or 2B) the location of the depth-sensing sensor 102, although it is not implemented. A machine representation 208 is placed within the central region 206 to provide the approximate size and location of a machine (e.g., vehicle 500) in relation to obstacles for reference. A series of concentric circles 210 are also shown, which may indicate that information is available about this area but no obstacles have been detected. If an obstacle is detected, the obstacle may be located within the concentric circles 210, and a blocked area 212 (indicated by the empty margin) may exist behind the obstacle boundary line 204 (from the viewpoint of the depth-sensing sensor 102). When the obstacle and / or the depth-sensing sensor 102 move relative to each other, thereby removing the blockage, the blocked object may become visible in additional iterations of the process. The obstacle boundary line 204 in Figure 2B has moved relative to the obstacle boundary line in Figure 2A.

[0021] System 100 may include a tracking state manager 112. The tracking state manager 112 may create and update various tracking states identified by the object detector 110. The tracking states may indicate an obstacle tracked over time during subsequent iterations of the method described herein. The tracking states may be used to determine the speed and direction of the obstacle (relative to the machine, the ground, or any other reference frame).

[0022] Since LiDAR data does not directly measure the velocity and direction of obstacles, embodiments of the present disclosure can use a Kalman filter to generate refined estimates of the velocity and direction of obstacles, which may include other variables related to the obstacles. In some embodiments of the present disclosure, each of one or more tracked object states is generated and / or refined using a Kalman filter.

[0023] A Kalman filter, also known as a linear quadratic estimation (LQE), generates an estimate of some unknown variable using a set of inaccurate and / or indirect measurements over time. In embodiments of this disclosure, the estimated unknown variable may be the velocity of a detected obstacle, which may include directional and / or velocity information. The Kalman filter may include a predictive operation that can generate an estimate of the unknown variable based on the inaccurate and / or indirect measurements. Secondly, the Kalman filter estimate may be updated when a new set of inaccurate and / or indirect measurements is received. In embodiments, more weights may be given to data and / or estimates with higher certainty. The Kalman filter may include, or may be, any modified form, such as an extended Kalman filter or an unscented Kalman filter.

[0024] System 100 may include a velocity estimator 114. The velocity estimator can determine the estimated velocity of a detected object corresponding to one or more tracking states. The velocity estimator 114 can determine the estimated velocity of a newly detected obstacle boundary line based on hypothetical correlations with one or more existing tracking states. The estimated velocity may include at least one of magnitude or directional components. In some instances, the determination of which of the tracking states (if any) corresponds to the current obstacle boundary line may be based on the magnitude component of the velocity, the directional component of the velocity, or both.

[0025] The velocity estimator 114 can determine the estimated velocity by comparing the current detection of an object determined using one or more LiDAR sensors, the previous detection of that object or another object determined using one or more LiDAR sensors, and the elapsed time between the current detection and the previous detection. The physical distance between the two locations of each detection, divided by the elapsed time, may give the magnitude and / or direction of the estimated velocity. The velocity estimator 114 may pass the potential distance and / or estimated velocity via a gating function. The gating function may simply allow for additional calculations and / or considerations when the distance, velocity, or other attributes are below a certain threshold. The gating function can prevent unnecessary calculations related to potential matches that are unlikely to actually be related. In other embodiments, the gating function may be applied after the estimated velocity has been determined and before comparisons to determine correlations.

[0026] The velocity estimator 114 may include a coordinate converter. The coordinate converter can transform one or more tracked object states into a coordinate system corresponding to the current detection of the object, where the comparison of the estimated velocity with one or more tracked object states is in that coordinate system. When comparing two—for example, continuous—frames (e.g., comparing two continuous point clouds), one frame (e.g., the previous frame) can be transformed or modified into the coordinate system of the subsequent or current frame (or vice versa) to compensate for ego motion. By compensating for ego motion, the motion of the ego vehicle is excluded, so the analysis only needs to describe the motion of the object relative to the ego vehicle—therefore enabling alignment of two frames (e.g., two LiDAR point clouds) in the same coordinate system.

[0027] After the estimated velocity is determined, the tracking state manager 112 (or other components) can determine which of the tracking states corresponds to the current object boundary line and update such tracking states with the new information using a Kalman filter accordingly. The tracking state manager 112 can compare the estimated velocity with one or more tracking object states corresponding to one or more previously detected objects.

[0028] Looking at Figure 3, the tracking state manager 112 can create and / or analyze a probability distribution corresponding to the velocity of each detected object among previously detected objects. An exemplary probability distribution diagram 300 is shown in Figure 3. Generally, the probability distribution diagram 300 has a probability axis 302 (e.g., weights of the distribution) and a value axis 304. It should also be understood that some embodiments of this disclosure may use higher-dimensional (e.g., 3D) considerations, and that Figure 3 presents a simplified probability distribution diagram 300 for the reader's understanding.

[0029] A first probability distribution 306 and a second probability distribution 308 are shown on probability distribution diagram 300. A measurement value 310 is shown at a specific location related to the first probability distribution 306 and the second probability distribution 308. Embodiments of this disclosure determine which of the probability distributions 306 and 308, if any, is most likely to correspond to the measurement value 310.

[0030] A first probability distribution 306 may have an associated first gating function 312. A relatively broad first probability distribution 306 may have a correspondingly broad gating function, indicating that a wide variety of measurement results can correspond to the first probability distribution. A second probability distribution 308 may have an associated second gating function 314. A relatively sharp second probability distribution 308 may have a correspondingly narrow gating function, indicating that measurement results with limited diversity can correspond to the second probability distribution. As seen in the example in Figure 3, the measurement 310 lies within both the first gating function 312 and the second gating function 314. If the measurement 310 lies outside one or more gating functions, the corresponding probability distribution is no longer considered.

[0031] In the example in Figure 3, since the measurement 310 passes through both gating functions 312 and 314 (for example, lies inside them), the tracking state analyzer 112 (or other component) can determine the probability distribution (representing an object detected before one of the objects detected before a set) that has the highest probability (or lies within it) corresponding to the measurement 310 (representing the current obstacle boundary line). The tracking state analyzer 112 can analyze a certain distance between the measurement 310 and the respective probability distributions 306 and 308. In some embodiments of the present disclosure, a certain distance may be based on the highest probability, represented by the height of the probability distribution associated with that measurement. Thus, since the measurement 310 is likely closest to the second probability distribution 308, as shown, the tracking state analyzer 112 can select the second probability distribution as the most likely to correspond to the measurement 310.

[0032] In other embodiments of the present disclosure, a certain distance may be the Mahalanobis distance as a cost function. However, the Mahalanobis distance may be more favorable to a wider range of probability distributions, because it may be smaller for the first probability distribution 306. Therefore, in embodiments of the present disclosure, the Mahalanobis distance may be used for the gating function, while the choice is based on the possibilities (e.g., the vertical height of each probability distribution).

[0033] After the corresponding tracking state has been identified, the tracking state manager 112 (or other component) can generate an updated tracking object state by updating the tracking object state corresponding to the previously detected object, which has the highest probability corresponding to the object, using the estimated velocity. The updated tracking state may be referred to as the refined tracking state. The tracking state manager 112 may use a Kalman filter when creating the refined tracking state.

[0034] System 100 may include a vehicle controller 116. Referring to Figure 1A, System 100 may include a vehicle controller 116 that can analyze information in the tracking state to determine obstacles in the physical environment around the depth perception sensor 102. The vehicle controller 116 can then instruct or transmit information related to one or more vehicle actions based on the determined obstacles, such as applying the brakes or changing the direction of the vehicle. For example, the vehicle controller 116 may run (or be otherwise related to) an autonomous driving software stack that may include a perception layer, a world model management layer, a planning layer, a control layer, an actuation layer, an obstacle avoidance layer, and / or one or more other layers.

[0035] System 100 may be associated with a computing system 600 within an autonomous vehicle 500, as illustrated and described with respect to Figures 5A-5D and 6. For example, an autonomously operating autonomous vehicle 500 may include a computer system 600 that performs numerous observations around obstacles and determines a number of actions that the vehicle 500 will perform to avoid those obstacles.

[0036] Referring here to Figure 1B, each block of Method 150 as described herein may include a computing process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory. Method 150 may also be performed as computer-available instructions stored in a non-temporary computer storage medium. Method 150 may be provided, to name a few, as a standalone application, service, or hosted service (standalone or in combination with another hosted service), or as a plug-in to another product. In addition, Method 150 is described, for example, with respect to System 100 in Figure 1A and / or Computer System 600 in Figure 6. However, this method may be performed by any one system or any combination of systems, including but not limited to those described herein, in addition or alternatively.

[0037] Figure 1B is a flowchart illustrating Method 150 for detecting and tracking objects among subsequent sets of LiDAR data (or other depth-perceiving data) and refining those detections over time, according to some embodiments of the present disclosure. Method 150 includes, in block B102, obtaining a current detection from, for example, a depth-perceiving sensor 102 and / or a sensor controller 104. Method 150 may also include, in block B102, determining one or more object boundaries from the LiDAR data, where the object boundaries can define one or more objects as detected in the LiDAR data, as shown in Figures 2A-B and discussed herein. The current detection (current detection A in Figure 1B) may be received from the depth-perceiving sensor 102 (e.g., a LiDAR sensor), where the current detection represents a sweep or other capture of the depth-perceiving sensor 102 at a particular time. One or more object boundaries in the current detection may be associated with a particular time for determining estimated velocity.

[0038] Method 150 includes, in block B104, comparing one or more object boundaries of the current detection with one or more previous detection and / or tracking states. A first tracking state (tracking state 1 in Figure 1B) and a second tracking state (tracking state 2 in Figure 1B) may be considered in relation to the object boundaries of the current detection, respectively. The first and second tracking states may also be potential candidates that are interrelated with the object boundaries of the current detection.

[0039] In embodiments of the present disclosure, Method 150 includes comparing the current obstacle boundary line in block B104 with a first obstacle boundary line corresponding to a first tracking state for generating a first velocity estimate and a second obstacle boundary line corresponding to a second tracking state for generating a second velocity estimate, using an Iterative Nearest (ICP) algorithm. The Iterative Nearest (ICP) algorithm can identify offset vectors such that the offset vector combined with the first tracking state matches the second tracking state. In embodiments, the ICP algorithm can calculate offset vectors indicating the magnitude and direction of the obstacle's motion (if any). Various distance and velocity measurements can be calculated based on the offset vectors.

[0040] An example of the ICP algorithm is shown in Figure 4, where the ego vehicle 400 is moving relative to the parked vehicle 402. A series of four detections 404 (shown sequentially) are captured as the ego vehicle 400 passes the parked vehicle 402. A corresponding set of four obstacle boundary lines 406A-D are detected. Detection 1 (acquired when the ego vehicle 400 is far from and approaching the parked vehicle 402) includes a first obstacle boundary line 406A, which is a straight line indicating the first edge of the parked vehicle 402. The centroid 408A of detection 1 is at the center of this straight line. Detection 2 (acquired when the ego vehicle 400 is close to and approaching the parked vehicle 402) includes a second obstacle boundary line 406B, which is "L" shaped and indicates a portion of the parked vehicle 402 visible to the depth-perceiving sensor 102 relative to the ego vehicle 400. The centroid 408B of detection 2 is at the average position of the "L" shape. The ICP algorithm can provide a more complete obstacle boundary by relating the first obstacle boundary line 406A to the second obstacle boundary line 406B. In relation to detection 2, the first obstacle boundary line 406A is shown as a dashed line relative to the second obstacle boundary line 406B. Detection 3 (acquired when the ego vehicle 400 is close to and passing the parked vehicle 402) includes a third obstacle boundary line 406C which is "L" shaped and shows a portion of the parked vehicle 402 visible to the depth-perceiving sensor 102 of the ego vehicle 400. The centroid 408C of detection 3 is at the average position of the "L" shape. The ICP algorithm can provide a more complete obstacle boundary by associating a first obstacle boundary 406A and / or a second obstacle boundary 406B (or a combined obstacle boundary that shows at least partially both the first obstacle boundary 406A and the second obstacle boundary 406B) with a third obstacle boundary 406C. Detection 4 (acquired when the ego vehicle 400 passes the parked vehicle 402) includes a fourth obstacle boundary 406D, which is a straight line indicating the second edge of the parked vehicle 402. The centroid 408D of detection 4 is at the center of this straight line.The ICP algorithm can provide a more complete obstacle boundary by associating the first obstacle boundary 406A, the second obstacle boundary 406B, and / or the third obstacle boundary 406D (or a combined obstacle boundary showing at least partially any combination thereof) with the fourth obstacle boundary 406D. It should be understood that detections 3 and 4 are acquired when the ego vehicle 400 passes the parked vehicle 402 and include substantially reflected obstacle boundary (this is for illustrative purposes only) of detections 2 and 1, respectively. Typically, each detection will occur at a time interval rather than at a relative position to a particular obstacle. Figure 4 shows each detection for illustrative purposes only.

[0041] It should be understood that the prior art system used only the center of gravity 408A-D to determine the position of the obstacle in each iteration (e.g., detections 1-4). As shown in Figure 4, the center of gravity 408A-D moves while the ego vehicle 400 passes over the parked vehicle 402. This resulted in the detection of motion in the prior art system. The ICP algorithm reduces or eliminates the perceived motion of the parked vehicle 402 by comparing each obstacle boundary line 406A-D instead of relying on the center of gravity 408A-D. The ICP algorithm can be used to calculate an offset vector for aligning two or more obstacle boundary lines 406A-D. For the parked vehicle 402 in Figure 4, the offset vector will be zero or nearly zero, resulting in little or no detected motion of the parked vehicle. For a moving vehicle (not shown in Figure 4), the translation vector would indicate the direction and magnitude of the vehicle's motion. The magnitude of the motion can be determined at least partially based on the translation of the obstacle between each of the two detections, divided by the time interval between the detections. The direction of the offset vector can be determined at least partially based on the direction of translation.

[0042] In embodiments of this disclosure, the comparison of the current obstacle detection with one or more previous detections is based at least in part on the similarity of the shapes of the respective obstacles, since the shapes are unlikely to change dramatically between single frames of LiDAR data. A first obstacle boundary line may include a first shape metric indicating a first geometric shape, and the current obstacle boundary line may include a second shape metric indicating a second geometric shape. Thus, the determination that the current obstacle boundary line most closely corresponds to the first obstacle boundary line may further be based at least in part on the first and second geometric shapes.

[0043] A comparison of the previously mentioned first object boundary line of the current detection with the second object boundary line of the tracking state can be performed, at least partially, using the ICP algorithm. Although the ICP position change is dramatic in the example in Figure 4, in embodiments of this disclosure, consecutive LiDAR data sets can be much closer together. Therefore, the difference in the shape of the obstacle boundary lines 406A-D between two consecutive scans of LiDAR data can be minimal, and can provide an accurate measurement of the motion of the physical object between consecutive scans. In embodiments of this disclosure, it should be understood that the ICP algorithm can describe one or more tracking object states or transform them into a coordinate system corresponding to the current detection of the object in other ways, and the comparison of the estimated velocity with one or more tracking object states may be in the updated coordinate system. This transformation to the updated coordinate system may be based on the measured motion of the ego machine 500 (e.g., ego motion).

[0044] Method 150 includes determining in block B106 whether a correlation exists between the current detection and any of the previous detections and / or tracking states. This operation may be performed by determining a cost function (or other probability distribution) associated with the estimated velocity of each potential correlation. The estimated velocity of the potential match may be determined, and then a cost function indicating the likelihood of a correlation may be calculated based at least in part on the estimated velocity. The estimated velocity of an object may be determined at least in part on the current detection of the object determined using one or more LiDAR sensors, the previous detection of that object or another object determined using one or more LiDAR sensors, and a comparison of the elapsed time between the current detection and the previous detection. Both the current detection of an object and the previous detection of that object may use ICP as the reference position, and the previous detection of that object may be adjusted by coordinate transformation.

[0045] Method 150 may further include, in block B106, comparing the estimated velocity to one or more tracked object states corresponding to one or more previously detected objects, where each of the tracked object states represents a probability distribution corresponding to the velocity of each detected object among the previously detected objects. The probability distribution may indicate the likelihood that the estimated velocity corresponds to its tracked object state. Based on the analysis of the probability distribution, one of the previously detected objects may be identified as having or being among the highest probability of corresponding to an object.

[0046] Method 150 further includes, in block B106, calculating a cost function corresponding to a probability distribution. In the example in Figure 1B, current detection A may, when compared with tracking state 1, produce a first probability distribution, and when compared with tracking state 2, produce a second probability distribution. The cost function may be at least partially based on the corresponding probability distributions, as shown in Figure 3. The first cost function may be calculated using the first probability distribution and the first velocity estimate. The second cost function may be calculated using the second probability distribution and the second velocity estimate.

[0047] The cost function, if any, can be evaluated to determine which is most likely to correspond to the current obstacle boundary line. Typically, a lower cost function indicates a higher probability of correlation. Method 150 may further include determining in block B106 that the current obstacle boundary line most closely corresponds to the first obstacle boundary line, based on the fact that the first cost function is smaller than the second cost function.

[0048] When a correlation with a first tracking state is found, method 150 includes updating the first tracking state in block B108. Alternatively, when a correlation with a second tracking state is found, method 150 includes updating the second tracking state in block B110. Each updated tracking state may include information on the previous and current obstacle boundary lines that are found to correspond to each tracking state.

[0049] If no correlation is found, method 150 includes creating a new tracking state for the object in block B112. If no correlation is found with the previous tracking state, the newly discovered object may be in the field of view from a distance or from behind an obstruction. The new tracking state may then be used for further comparison to determine whether the subsequent current obstacle boundary line corresponds to the new tracking state. As such, the new tracking state may be refined in a subsequent sweep of the depth-perceiving sensor 102.

[0050] Method 150 includes applying a Kalman filter to a first tracking state in block B114, following block B108. Method 150 includes creating a refined first tracking state in block B116. The refined tracking state may be created by applying a Kalman filter to add additional information to the tracking state. The refined tracking state may (directly or indirectly) indicate the current obstacle detection in addition to one or more previous obstacle detections of the same physical obstacle. Similarly, Method 150 includes applying a Kalman filter to a second tracking state in block B118, following block B110. Method 150 includes creating a refined second tracking state in block B120.

[0051] Method 150 includes controlling the machine in block B120 at least in part on an elaborated and / or new tracking state. For example, a tracking state may represent a physical object that may directly or indirectly affect the autonomous vehicle 500 or other machines. Such an obstacle may be a moving object, such as another vehicle or pedestrian, or a stationary object, such as a building and a tree. An obstacle may have a position relative to the vehicle, a 3D motion vector (which may include acceleration, rotation, or other motion instructions), and a relative size (based on how many pixels correspond to the obstacle). In some embodiments, based at least in part on information about an obstacle, the system 100 may determine the likelihood that the obstacle may affect the vehicle, and the system may determine one or more remedial actions to be performed to avoid the obstacle. For example, the vehicle controller 118 may determine that the vehicle should brake, change direction, or accelerate to avoid the obstacle. In instances where an obstacle cannot be avoided, system 100 may determine that one or more remedial actions should be taken, such as minimizing damage or activating other safety features. Control of block B120 may be performed to avoid the identified obstacle by taking one or more remedial actions. Vehicle control may include sending commands to any of several vehicle systems, such as those described with respect to Figures 5A-5D.

[0052] Exemplary autonomous vehicle Figure 5A shows an exemplary autonomous vehicle 500 according to some embodiments of the present disclosure. The autonomous vehicle 500 (as otherwise referred to herein as "Vehicle 500") may include, but is not limited to, passenger vehicles such as cars, trucks, buses, first responder vehicles, shuttles, electric or motorized bicycles, motorcycles, fire engines, police vehicles, ambulances, boats, construction vehicles, submarines, drones, trailer-mounted vehicles, and / or other types of vehicles (e.g., unmanned and / or carrying one or more passengers). Autonomous vehicles are generally described in terms of automation levels as defined by the National Highway Traffic Safety Administration (NHTSA), departments within the U.S. Department of Transportation, and the Society of Automotive Engineers (SAE) "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicle" (standard number J3016-201806, published June 15, 2018; standard number J3016-201609, published September 30, 2016; and previous and future versions of this standard). Vehicle 500 may have the capability to perform functions at one or more of the autonomous driving levels from Level 3 to Level 5. For example, depending on the embodiment, Vehicle 500 may have the capability of conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5).

[0053] Vehicle 500 may include components such as the vehicle's chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components. Vehicle 500 may include a propulsion system 550, such as an internal combustion engine, a hybrid power unit, a fully electric engine, and / or another propulsion system type. The propulsion system 550 may be connected to the vehicle's drivetrain, which may include a transmission, to enable propulsion for the vehicle 500. The propulsion system 550 may be controlled in response to receiving signals from a throttle / accelerator 552.

[0054] A steering system 554, which may include a steering wheel, may be used to steer the vehicle 500 (for example, along a desired course or route) when the propulsion system 550 is operating (for example, when the vehicle is moving). The steering system 554 may receive signals from a steering actuator 556. The steering wheel may also be an option for fully automated (level 5) functionality.

[0055] The brake sensor system 546 may be used to operate the vehicle brakes in response to receiving signals from the brake actuator 548 and / or the brake sensor.

[0056] The controller 536, which may include one or more system-on-a-chip (SoC) 504 (Figure 5C) and / or GPUs, can provide signals (e.g., expressions of commands) to one or more components and / or systems of the vehicle 500. For example, the controller can send signals to actuate the vehicle brakes via one or more brake actuators 548, actuate the steering system 554 via one or more steering actuators 556, and actuate the propulsion system 550 via one or more throttle / accelerators 552. The controller 536 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operational commands (e.g., signals representing commands) to enable autonomous driving and / or assist the driver in driving the vehicle 500. The controller 536 may include a first controller 536 for autonomous driving functions, a second controller 536 for functional safety functions, a third controller 536 for artificial intelligence functions (e.g., computer vision), a fourth controller 536 for infotainment functions, a fifth controller 536 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 536 may handle two or more of the aforementioned functions, and two or more controllers 536 may handle a single function, and / or any combination thereof.

[0057] The controller 536 can provide signals for controlling one or more components and / or systems of the vehicle 500 in response to sensor data (e.g., sensor inputs) received from one or more sensors. Sensor data may be received from, for example, and without limitation, global navigation satellite system sensors 558 (e.g., global positioning system sensors), RADAR sensors 560, ultrasonic sensors 562, LIDAR sensors 564, inertial measurement unit (IMU) sensors 566 (e.g., accelerometers, gyroscopes, magnetic compasses, magnetometers, etc.), microphones 596, stereo cameras 568, wide-view cameras 570 (e.g., fisheye cameras), infrared cameras 572, surround cameras 574 (e.g., 360-degree cameras), long-range and / or medium-range cameras 598, speed sensors 544 (e.g., for measuring the speed of a vehicle 500), vibration sensors 542, steering sensors 540, brake sensors (e.g., as part of a brake sensor system 546), and / or other sensor types.

[0058] One or more of the controllers 536 may receive inputs (represented, for example, by input data) from the instrument cluster 532 of the vehicle 500 and provide outputs (represented, for example, by output data, display data, etc.) via a human-machine interface (HMI) display 534, an audible annunciator, a loudspeaker, and / or other components of the vehicle 500. The outputs may include information such as vehicle velocity, speed, time, map data (e.g., HD map 522 in Figure 5C), location data (e.g., the location of the vehicle 500, such as on the map), direction, the location of other vehicles (e.g., occupied grid), and information about objects and the status of objects as perceived by the controller 536. For example, the HMI display 534 may display information regarding the presence of one or more objects (e.g., road signs, warning signs, changes in traffic signals, etc.) and / or driving operations that the vehicle has performed, is performing, or could perform (e.g., that it is now changing lanes, or that it is about to exit at Exit 34B within 3.22 km (2 miles)).

[0059] Vehicle 500 further includes a network interface 524 that can communicate over one or more networks using one or more wireless antennas 526 and / or a modem. For example, the network interface 524 may have the capability to communicate over LTE, WCDMA®, UMTS, GSM, CDMA2000, etc. The wireless antennas 526 can also enable communication between objects in the environment (e.g., vehicles, mobile devices, etc.) using local area networks such as Bluetooth®, Bluetooth® LE, Z-Wave, ZigBee, and / or low-power wide-area networks (LPWANs) such as LoRaWAN, SigFox.

[0060] Figure 5B shows examples of camera positions and fields of view of the exemplary autonomous vehicle 500 of Figure 5A, according to several embodiments of the present disclosure. The cameras and their respective fields of view are exemplary embodiments and are not intended to limit the scope. For example, additional and / or alternative cameras may be included, and / or cameras may be placed in different locations on the vehicle 500.

[0061] The camera type may include, but is not limited to, a digital camera that can be used with components and / or systems of the vehicle 500. The camera may operate at Automotive Safety Integrity Level (ASIL) B and / or other ASILs. Depending on the embodiment, the camera type may have the capability of any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc. The camera may have the capability to use a roll shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include an RCCC (red clear clear clear) color filter array, an RCCB (red clear clear blue) color filter array, an RBGC (red blue green clear) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, clear pixel cameras, such as cameras having RCCC, RCCB, and / or RBGC color filter arrays, may be used in efforts to increase light sensitivity.

[0062] In some applications, one or more cameras may be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function mono-camera may be installed to provide functions including lane departure warning, traffic sign assist, and intelligent headlamp control. One or more cameras (e.g., all cameras) may simultaneously record and provide image data (e.g., video).

[0063] One or more of the cameras may be mounted in custom-designed (3D-printed) mounting parts to eliminate stray light and reflections from inside the vehicle (e.g., reflections from the dashboard reflected in the windshield mirror) that may interfere with the camera's image data capture capability. Referring to side mirror mounting parts, the side mirror parts may be custom 3D-printed so that the camera mounting plate conforms to the shape of the side mirror. In some examples, the camera may be integrated within the side mirror. For side-view cameras, the camera may also be integrated within four struts located at each corner of the cabin.

[0064] A camera having a field of view that includes a portion of the environment in front of the vehicle 500 (e.g., a forward-facing camera) may be used for surround view to help identify the forward path and obstacles and, with the help of one or more controllers 536 and / or control SoCs, to help provide information essential for generating an occupied grid and / or determining a preferred vehicle path. The forward-facing camera may also be used to perform many of the same ADAS functions as LIDAR, including emergency braking, pedestrian detection, and collision avoidance. The forward-facing camera may also be used for ADAS functions and systems, including other functions such as Lane Departure Warning ("LDW"), Autonomous Cruise Control ("ACC"), and / or traffic sign recognition.

[0065] Various cameras may be used in forward-facing configurations, including, for example, a monocular camera platform that includes a CMOS (complementary metal oxide semiconductor) color imaging device. Another example may be a wide-view camera 570, which can be used to capture objects entering the view from the periphery (e.g., pedestrians, crossing traffic, or bicycles). Although only one wide-view camera is shown in Figure 5B, any number of wide-view cameras 570 may be present in the vehicle 500. In addition, long-range cameras 598 (e.g., a long-view stereo camera pair) may be used for depth-based object detection, particularly for objects for which the neural network has not yet been trained. Long-range cameras 598 may also be used for object detection and classification, as well as basic object tracking.

[0066] One or more stereo cameras 568 may also be included in the forward-facing configuration. The stereo camera 568 may include an integrated control unit with an expandable processing unit that may provide a programmable logic (FPGA) and a multi-core microprocessor with an integrated CAN or Ethernet® interface on a single chip. Such a unit may be used to generate a 3D map of the vehicle's environment, including distance estimates of all points in the image. An alternative stereo camera 568 may include a compact stereo vision sensor that includes two camera lenses (one on the left and one on the right) and an image processing chip that can measure the distance from the vehicle to an object and use the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 568 may be used in addition to or instead of those described herein.

[0067] A camera having a field of view including a portion of the environment on the sides of the vehicle 500 (e.g., a side-view camera) may be used for surround view, providing information used to create and update the occupancy grid and generate side impact collision warnings. For example, surround cameras 574 (e.g., four surround cameras 574 as shown in Figure 5B) may be positioned on the vehicle 500. The surround cameras 574 may include wide-view cameras 570, fisheye cameras, 360-degree cameras, and / or similar. For example, four fisheye cameras may be positioned in front of, behind, and on the sides of the vehicle. In an alternative configuration, the vehicle may use three surround cameras 574 (e.g., left, right, and rear) and utilize one or more other cameras (e.g., forward-facing cameras) as a fourth surround view camera.

[0068] A camera having a field of view that includes a portion of the environment behind the vehicle 500 (e.g., a rear-view camera) may be used for parking assistance, surround view, rear collision warning, and creation and updating of the occupancy grid. A wide variety of cameras may be used, including, but not limited to, cameras suitable as forward-facing cameras (e.g., long-range and / or medium-range cameras 598, stereo cameras 568), infrared cameras 572, etc., as described herein.

[0069] Figure 5C is a block diagram of an exemplary system architecture of the exemplary autonomous vehicle 500 of Figure 5A, according to some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted together. Furthermore, many of the elements described herein are functional entities that can be implemented as individual or distributed components or in combination with other components, and in any appropriate combination and location. The various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory.

[0070] Each component, feature, and system of vehicle 500 in Figure 5C is illustrated as being connected via bus 502. Bus 502 may include a Controller Area Network (CAN) data interface (or referred to as the "CAN bus"). CAN may also be a network within vehicle 500 used to help control various features and functions of vehicle 500, such as the operation of brakes, acceleration, steering, windshield wipers, etc. The CAN bus may be configured to have dozens or hundreds of nodes, each having its own unique identifier (e.g., CAN ID). The CAN bus may be read to find steering angle, ground speed, engine revolutions per minute (RPM), button position, and / or other vehicle status indicators. The CAN bus may be ASIL B compliant.

[0071] Bus 502 is described herein as a CAN bus, but this is not intended to limit it. For example, in addition to or as an alternative to a CAN bus, FlexRay and / or Ethernet® may be used. In addition, a single line is used to represent bus 502, but this is not intended to limit it. There may be any number of buses 502, which may include, for example, one or more CAN buses, one or more FlexRay buses, one or more Ethernet® buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 502 may be used to perform different functions and / or for redundancy. For example, a first bus 502 may be used for collision avoidance functions, and a second bus 502 may be used for operation control. In any example, each bus 502 may communicate with any of the components of the vehicle 500, and two or more buses 502 may communicate with the same component. In some examples, each SoC 504, each controller 536, and / or each computer within the vehicle may have access to the same input data (e.g., input from sensors in the vehicle 500) and may be connected to a common bus such as a CAN bus.

[0072] The vehicle 500 may include one or more controllers 536, such as those described herein with respect to Figure 5A. The controllers 536 may be used for a variety of functions. The controllers 536 may be connected to any of the various other components and systems of the vehicle 500 and may be used for the control of the vehicle 500, the artificial intelligence of the vehicle 500, infotainment for the vehicle 500, and / or the like.

[0073] Vehicle 500 may include a system-on-a-chip (SoC) 504. The SoC 504 may include a CPU 506, a GPU 508, a processor 510, a cache 512, an accelerator 514, a data store 516, and / or other components and features not shown. The SoC 504 may be used to control vehicle 500 in various platforms and systems. For example, the SoC 504 may be coupled in a system (e.g., a system of vehicle 500) that has an HD map 522 that can obtain map refreshes and / or updates via a network interface 524 from one or more servers (e.g., server 578 in Figure 5D).

[0074] The CPU 506 may include a CPU cluster or CPU complex (also referred to as "CCPLEX"). The CPU 506 may include multiple cores and / or L2 caches. For example, in some embodiments, the CPU 506 may include eight cores in a coherent multiprocessor configuration. In some embodiments, the CPU 506 may include four dual-core clusters, each cluster having its own dedicated L2 cache (e.g., 2MBL2 cache). The CPU 506 (e.g., CCPLEX) may be configured to support concurrent cluster operation, allowing any combination of the CPU 506 clusters to be active at any given time.

[0075] The CPU506 can implement power management capabilities that include one or more of the following features: individual hardware blocks may be automatically clock-gated when idle to conserve dynamic power; each core clock may be gated when a core is not actively executing instructions by executing WFI / WFE instructions; each core may be independently power-gated; each core cluster may be independently clock-gated when all cores are clock-gated or power-gated; and / or each core cluster may be independently power-gated when all cores are power-gated. The CPU506 can further implement enhanced algorithms for managing power states, where acceptable power states and expected wake-up times are specified, and the hardware / microcode determines the best power state to input to the cores, clusters, and CCPLEX. The processing core may support a simplified power state input sequence in software where the work is offloaded to the microcode.

[0076] The GPU508 may include an integrated GPU (or, as referred to herein, "iGPU"). The GPU508 may be programmable and efficient for parallel workloads. In some embodiments, the GPU508 may be able to use an enhanced tensor instruction set. The GPU508 may include one or more streaming microprocessors, each of which may include an L1 cache (e.g., an L1 cache with a storage capacity of at least 96KB), and two or more of the streaming microprocessors may share a cache (e.g., an L2 cache with a storage capacity of 512KB). In some embodiments, the GPU508 may include at least eight streaming microprocessors. The GPU508 may be able to use a Computation Application Programming Interface (API). In addition, the GPU508 may be able to use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0077] The GPU508 can be power-optimized for optimal performance in automotive and embedded use cases. For example, the GPU508 can be manufactured on a FinFET (Fin field-effect transistor). However, this is not intended to be a limitation, and the GPU508 can be manufactured using other semiconductor manufacturing processes. Each streaming microprocessor can incorporate several mixed-precision processing cores divided into multiple blocks. Not limited to, for example, 64 PF32 cores and 32 PF64 cores may be divided into four processing blocks. In such an example, each processing block may be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, 2 mixed-precision NVIDIA tensor cores for deep learning matrix operations, an L0 instruction cache, a warp scheduler, a dispatch unit, and / or a 64KB register file. In addition, the streaming microprocessor may include independent parallel integer and floating-point data paths to provide efficient execution of workloads with a mixture of computation and addressing operations. A streaming microprocessor may include independent thread scheduling capabilities to enable finer-grained synchronization and coordination between concurrent threads. A streaming microprocessor may also include a combined L1 data cache and shared memory unit to simplify programming while improving performance.

[0078] In some examples, the GPU508 may include high-bandwidth memory (HBM) and / or a 16GB HBM2 memory subsystem to provide a peak memory bandwidth of 900 GB / s. In some examples, in addition to or instead of HBM memory, synchronous graphics random-access memory (SGRAM), such as graphics double data rate type five synchronous random-access memory (GDDR5), may be used.

[0079] The GPU508 can incorporate unified memory technology, including access counters, to enable more precise movement of memory pages to the processor that most frequently accesses them, thereby improving the efficiency of shared memory ranges across processors. In some examples, address translation service (ATS) support may be used to allow the GPU508 to directly access the CPU506 page table. In such examples, when the GPU508 memory management unit (MMU) experiences a miss, an address translation request may be sent to the CPU506. In response, the CPU506 can examine its page table for virtual-to-real-address mapping and send the translation back to the GPU508. As such, unified memory technology can enable a single, unified virtual address space for both the CPU506 and GPU508 memory, thereby simplifying GPU508 programming and porting of applications to the GPU508.

[0080] In addition, the GPU508 may include access counters that can record how often the GPU508 accesses the memory of other processors. Access counters can help ensure that memory pages are moved to the physical memory of the processor that accesses that page most frequently.

[0081] The SoC504 may include any number of caches 512, including those described herein. For example, cache 512 may include an L3 cache available to both the CPU 506 and the GPU 508 (e.g., connected to both the CPU 506 and the GPU 508). Cache 512 may include a write-back cache that can record line states, for example, by using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). The L3 cache may include 4MB or more, depending on the embodiment, although a smaller cache size may be used.

[0082] The SoC504 may include an arithmetic logic unit (ALU) that can be used to perform processing for any of the various tasks or operations of the vehicle 500 (for example, a processing DNN). In addition, the SoC504 may include a floating-point unit (FPU) (or other mass coprocessor or numerical coprocessor type) for performing mathematical operations within the system. For example, the SoC104 may include one or more FPUs integrated as execution units within the CPU506 and / or GPU508.

[0083] The SoC504 may include one or more accelerators 514 (e.g., a hardware accelerator, a software accelerator, or a combination thereof). For example, the SoC504 may include a hardware acceleration cluster that may include an optimized hardware accelerator and / or a large on-chip memory. The large on-chip memory (e.g., 4 MB of SRAM) may enable the hardware acceleration cluster to accelerate neural networks and other computations. The hardware acceleration cluster may be used to complement the GPU508 and to offload some of the tasks of the GPU508 (e.g., to free up more cycles of the GPU508 to perform other tasks). As an example, accelerator 514 may be used for target workloads that are sufficiently stable to be suitable for acceleration (e.g., perception, convolutional neural networks (CNNs), etc.). In this specification, the term "CNN" may include all types of CNNs, including region-based or regional convolutional neural networks (RCNNs) and fast RCNNs (for example, as used for object detection).

[0084] The accelerator 514 (e.g., a hardware acceleration cluster) may include a deep learning accelerator (DLA). The DLA may include one or more tensor processing units (TPUs) that can be configured to provide an additional 10 trillion operations per second for deep learning applications and inference. The TPU may also be an accelerator configured and optimized to perform image processing functions (e.g., CNN, RCNN, etc.). The DLA may further be optimized for a specific set of neural network types and floating-point operations, as well as for inference. The design of the DLA can provide more performance per millisecond than a general-purpose GPU and significantly exceed the performance of a CPU. The TPU can perform several functions, including, for example, single-instance convolutional functions supporting INT8, INT16, and FP16 data types for both features and weights, as well as post-processing functions.

[0085] DLA can quickly and efficiently run neural networks, particularly CNNs, on processed or unprocessed data for any of a variety of functions, including but not limited to: CNNs for object recognition and detection using data from camera sensors; CNNs for distance estimation using data from camera sensors; CNNs for emergency vehicle detection, identification, and detection using data from microphones; CNNs for facial recognition and vehicle owner identification using data from camera sensors; and / or CNNs for security and / or safety-related events.

[0086] DLA can perform any function of GPU508, and by using inference accelerators, for example, a designer can target either DLA or GPU508 for any function. For example, a designer can focus on CNN and floating-point arithmetic processing on DLA, and leave other functions to GPU508 and / or other accelerators 514.

[0087] The accelerator 514 (for example, a hardware accelerator cluster) may include a programmable vision accelerator (PVA), which may also be referred to herein as a computer vision accelerator. A PVA may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. A PVA can provide a balance between performance and flexibility. For example, each PVA may include, but is not limited to, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors.

[0088] A RISC core can interact with an image sensor (for example, the image sensor of one of the cameras described herein), an image signal processor, and / or similar devices. Each RISC core may contain any amount of memory. Depending on the embodiment, a RISC core may use one of several protocols. In some examples, a RISC core can run a real-time operating system (RTOS). A RISC core may be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, a RISC core may include an instruction cache and / or tightly coupled RAM.

[0089] DMA can enable PVA components to access system memory independent of the CPU506. DMA can support any number of features used to bring optimizations to the PVA, including but not limited to supporting multidimensional addressing and / or circular addressing. In some examples, DMA can support up to six or more dimensions of addressing, which may include block width, block height, block depth, horizontal block stepping, vertical block stepping, and / or depth stepping.

[0090] A vector processor may also be a programmable processor that can be designed to efficiently and flexibly execute the programming of computer vision algorithms and provide signal processing capabilities. In some examples, a PVA may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, a DMA engine (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem can act as the primary processing engine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or vector memory (e.g., VMEM). The VPU core may include a digital signal processor, such as a single-instruction, multiple-data (SIMD), or very-long instruction word (VLIW) digital signal processor. A combination of SIMD and VLIW can increase throughput and speed.

[0091] Each vector processor may include an instruction cache and be linked to dedicated memory. As a result, in some examples, each vector processor may be configured to run independently of other vector processors. In other examples, the vector processors included in a particular PVA may be configured to use data parallelism. For example, in some embodiments, multiple vector processors included in a single PVA can run the same computer vision algorithm, but on different regions of an image. In other examples, the vector processors included in a particular PVA can run different computer vision algorithms simultaneously on the same image, or even run different algorithms sequentially on the image or parts of an image. In particular, any number of PVAs may be included in a hardware acceleration cluster, and any number of vector processors may be included in each PVA. In addition, a PVA may include additional error correction code (ECC) memory to enhance overall system safety.

[0092] The accelerator 514 (for example, a hardware accelerator cluster) may include a computer vision network on-chip and SRAM to provide high-bandwidth, low-latency SRAM for the accelerator 514. In some examples, the on-chip memory may include at least 4 MB of SRAM consisting of eight field-configurable memory blocks, which may be accessible by both the PVA and DLA, for example, and not limited to. Each pair of memory blocks may include an advanced peripheral bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory may be used. The PVA and DLA can access the memory via a backbone that provides the PVA and DLA with high-speed access to the memory. The backbone may include a computer vision network on-chip that interconnects the PVA and DLA to the memory (for example, using an APB).

[0093] A computer vision network on-chip may include an interface that determines whether both the PVA and DLA are activatable and enable signals before any control signals / addresses / data are transmitted. Such an interface can provide separate phases and separate channels for transmitting control signals / addresses / data, as well as burst-type communication for continuous data transfer. This type of interface may conform to ISO 26262 or IEC 61508 standards, but other standards and protocols may be used.

[0094] In some embodiments, the SoC504 may include a real-time ray tracing hardware accelerator, such as the one described in Patent Document 1. The real-time ray tracing hardware accelerator may be used to quickly and efficiently determine the location and size of objects (e.g., in a world model) to generate real-time visualization simulations for RADAR signal interpretation, acoustic propagation synthesis and / or analysis, SONAR system simulation, general wave propagation simulation, comparison to LIDAR data for localization and / or other functions, and / or other uses. In some embodiments, one or more tree traversal units (TTUs) may be used to perform one or more ray tracing-related operations.

[0095] The accelerator 514 (e.g., a hardware accelerator cluster) has diverse applications for autonomous driving. The PVA may also be a programmable vision accelerator that can be used in critical processing stages in ADAS and autonomous vehicles. The capabilities of the PVA are suitable for areas of algorithms that require predictable processing at low power and low latency. In other words, the PVA works well in semi-high density or high density normal computations, even on small data sets, where predictable execution time is required along with low latency and low power. Therefore, since the PVA is efficient in operation in object detection and integer computation, in relation to a platform for autonomous vehicles, the PVA is designed to run classic computer vision algorithms.

[0096] For example, according to one embodiment of this technology, PVA is used to perform computer stereo vision. While semi-global matching-based algorithms may be used in some examples, this is not intended to be a limitation. Numerous applications for Level 3-5 autonomous driving require motion estimation / stereo matching on the fly (e.g., SFM (structure from motion), pedestrian recognition, lane detection, etc.). PVA can perform computer stereo vision functions with input from two monocular cameras.

[0097] In some applications, PVA can be used to perform high-density optical flow by processing raw RADAR data (e.g., using 4D Fast Fourier Transform) to provide processed RADAR data. In other applications, PVA is used for flight depth processing, for example, by processing raw flight data to provide processed flight data.

[0098] DLA can be used to run any type of network to enhance control and driving safety, for example, a neural network that outputs a confidence value for each object detection. Such confidence values ​​can be interpreted as probabilities or as providing the relative "weight" of each detection compared to other detections. This confidence value allows the system to make further decisions about which detections should be considered true positives rather than false positives. For example, the system can set a confidence threshold and consider only detections that exceed the threshold as true positives. In an automatic emergency braking (AEB) system, a false positive detection would cause the vehicle to automatically apply the emergency brakes, which is obviously undesirable. Therefore, only the most confident detection should be considered as a trigger for the AEB. DLA can run a neural network that regresses on the confidence values. The neural network can accept at least a subset of parameters as its input, such as bounding box dimensions, ground plane estimation acquired (e.g., from another subsystem), vehicle orientation, distance, inertial measurement unit (IMU) sensor output correlated with 3D position estimation of an object acquired from the neural network and / or other sensors (e.g., LIDAR sensor 564 or RADAR sensor 560), and others.

[0099] The SoC504 may include a data store 516 (for example, memory). The data store 516 may also be the on-chip memory of the SoC504 and can store neural networks that will run on the GPU and / or DLA. In some examples, the data store 516 may have a capacity large enough to store multiple instances of the neural network for redundancy and safety. The data store 512 may include an L2 or L3 cache 512. References to the data store 516 may include references to memory associated with the PVA, DLA, and / or other accelerators 514, as described herein.

[0100] The SoC504 may include one or more processors 510 (e.g., integrated processors). The processors 510 may include a boot and power management processor, which may be a dedicated processor and subsystem for handling boot power and management capabilities and associated security enforcement. The boot and power management processor may also be part of the SoC504 boot sequence and can provide runtime power management services. The boot power and management processor may provide clock and voltage programming, assistance with system low-power state transitions, management of SoC504 thermal and temperature sensors, and / or management of SoC504 power states. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to temperature, and the SoC504 may use the ring oscillators to detect the temperatures of the CPU 506, GPU 508, and / or accelerator 514. If the temperature is determined to have exceeded a threshold, the boot and power management processor may enter a temperature fault routine, placing the SoC504 into a lower power state and / or putting the vehicle 500 into chauffeur safe shutdown mode (for example, safely shutting down the vehicle 500).

[0101] The processor 510 may further include a set of integrated processors that can perform the functions of an audio processing engine. The audio processing engine may also be an audio subsystem that enables full hardware support for multi-channel audio through multiple interfaces and a wide and flexible range of audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core having a digital signal processor with dedicated RAM.

[0102] The processor 510 may further include an always-on processor engine that can provide the necessary hardware features to support low-power sensor management and wake use cases. The always-on processor engine may include a processor core, tightly coupled RAM, support peripherals (e.g., timer and interrupt controllers), various I / O controller peripherals, and routing logic.

[0103] The processor 510 may further include a safety cluster engine, which includes a dedicated processor subsystem for handling safety management in automotive applications. The safety cluster engine may include two or more processor cores, tightly coupled RAM, supporting peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In safety mode, the two or more cores may operate in lockstep mode and function as a single core with comparison logic for detecting any differences between their operations.

[0104] The processor 510 may further include a real-time camera engine, which may include a dedicated processor subsystem for handling real-time camera management.

[0105] Processor 510 may further include a high dynamic range signal processor, which may include an image signal processor, a hardware engine that is part of the camera processing pipeline.

[0106] The processor 510 may include a video image synthesizer, which may also be a processing block (for example, implemented on a microprocessor) that implements post-video processing functions required by the video playback application to produce the final image for the player window. The video image synthesizer can perform lens distortion correction on the wide-view camera 570, the surround camera 574, and / or the in-cabin surveillance camera sensors. The in-cabin surveillance camera sensors are preferably monitored by a neural network running on another instance of the advanced SoC, configured to identify and respond appropriately to in-cabin events. The in-cabin system can perform lip-reading to activate cellular services and make phone calls, transcribe emails, change the vehicle's destination, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. Certain functions are available to the driver only when operating in autonomous mode and are otherwise disabled.

[0107] A video image synthesizer may include enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, if motion occurs in the video, noise reduction reduces the weight of information provided by adjacent frames and appropriately weights the spatial information. If the image or part of the image does not contain motion, the temporal noise reduction performed by the video image synthesizer can use information from previous images to reduce noise in the current image.

[0108] The video image synthesizer can also be configured to perform stereo rectification on the input stereo lens frame. Furthermore, the video image synthesizer can be used for user interface compositing when the operating system desktop is in use, so that the GPU508 is not required to continuously render new surfaces. Even when the GPU508 is powered on and actively performing 3D rendering, the video image synthesizer can be used to offload the GPU508 to improve performance and responsiveness.

[0109] The SoC504 may further include a Mobile Industry Processor Interface (MIPI) camera serial interface, a high-speed interface, and / or a video input block that can be used for camera and associated pixel input functions to receive video and input from a camera. The SoC504 may further include an input / output controller that can be controlled by software and can be used to receive I / O signals that are not committed to a specific role.

[0110] The SoC504 may further include a wide range of peripheral interfaces to enable communication with peripheral devices, audio codecs, power management, and / or other devices. The SoC504 may be used to process data from cameras (connected, for example, via Gigabit Multimedia Serial Link and Ethernet®), sensors (e.g., LiDAR sensor 564, RADAR sensor 560, etc., which may be connected via Ethernet®), data from bus 502 (e.g., vehicle speed, steering wheel position, etc.), and data from GNSS sensor 558 (connected, for example, via Ethernet® or CAN bus). The SoC504 may further include its own DMA engine and a dedicated high-performance mass storage controller which may be used to free up CPU 506 from routine data management tasks.

[0111] The SoC504 may also be an inter-terminal platform with a flexible architecture that spans automation levels 3-5, thereby providing a comprehensive functional safety architecture that leverages and efficiently uses computer vision and ADAS techniques for diversity and redundancy, and, together with deep learning tools, provides a platform for a flexible, reliable driving software stack. The SoC504 can be faster, more reliable, more energy-efficient, and more space-efficient than conventional systems. For example, when the accelerator 514 is coupled with the CPU 506, the GPU 508, and the data store 516 can provide a fast and efficient platform for autonomous vehicles at levels 3-5.

[0112] Therefore, this technology brings capabilities and functionality that cannot be achieved by conventional systems. For example, computer vision algorithms can be executed on a CPU, which can be configured using high-level programming languages ​​such as the C programming language to execute a wide variety of processing algorithms across a wide variety of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. Specifically, many CPUs cannot execute real-time complex object detection algorithms, which are required for in-vehicle ADAS applications and actual Level 3-5 autonomous vehicles.

[0113] In contrast to conventional systems, by providing a CPU complex, a GPU complex, and a hardware acceleration cluster, the technologies described herein enable multiple neural networks to run simultaneously and / or sequentially, and the results to be combined to enable Level 3–5 autonomous driving capabilities. For example, a DLA or a CNN running on a dGPU (e.g., GPU520) may include text and word recognition, enabling a supercomputer to read and understand traffic signs, including signs for which the neural network has not been specifically trained. The DLA may further include a neural network capable of identifying, interpreting, and providing a semantic understanding of signs and passing that semantic understanding to a route planning module running on the CPU complex.

[0114] As another example, multiple neural networks may run simultaneously, as required for Level 3, 4, or 5 driving. For instance, a warning sign consisting of a flashing light and the text "Caution: Flashing light indicates frozen conditions" may be interpreted independently or collectively by several neural networks. The sign itself may be identified as a traffic sign by a first deployed neural network (e.g., a trained neural network), and the text "Flashing light indicates frozen conditions" may be interpreted by a second deployed neural network that informs the vehicle's route planning software (preferably running on a CPU complex) that frozen conditions are present when flashing light is detected. The flashing light may be identified by informing the vehicle's route planning software of the presence (or absence) of the flashing light, and by operating a third deployed neural network through multiple frames. All three neural networks can run simultaneously within the DLA and / or on the GPU508, for example.

[0115] In some applications, a CNN for facial recognition and vehicle owner identification can use data from camera sensors to identify the presence of the legitimate driver and / or owner of vehicle 500. An always-on sensor processing engine may be used to unlock the vehicle and turn on the lights when the owner approaches the driver's side door, and in security mode, to stop the vehicle when the owner leaves the vehicle. In this way, SoC504 provides security against theft and / or vehicle hijacking.

[0116] In another example, a CNN for emergency vehicle detection and identification can detect and identify emergency vehicle sirens using data from microphone 596. In contrast to conventional systems that use a general classifier to detect sirens and manually extract features, SoC 504 uses a CNN for classifying environmental and urban sounds, as well as for classifying visual data. In a preferred embodiment, a CNN running on DLA is trained to identify the relative terminal velocity of emergency vehicles (for example, by using the Doppler effect). The CNN can also be trained to identify emergency vehicles specific to the local area in which the vehicle is operating, as identified by GNSS sensor 558. Thus, for example, when operating in Europe, the CNN may be able to detect European sirens, and when in the United States, the CNN may be able to identify only North American sirens. After an emergency vehicle is detected, a control program may be used, with the assistance of ultrasonic sensor 562, to perform emergency vehicle safety routines such as slowing down the vehicle, stopping it at the side of the road, parking the vehicle, and / or idling the vehicle until the emergency vehicle has passed.

[0117] The vehicle may include a CPU 518 (e.g., a separate CPU, or dCPU) which can be connected to the SoC 504 via a high-speed interconnect (e.g., PCIe). The CPU 518 may include, for example, an x86 processor. The CPU 518 may be used to perform any of a variety of functions, including, for example, mediating the consequences of a potential mismatch between ADAS sensors and the SoC 504, and / or monitoring the status and condition of the controller 536 and / or the infotainment SoC 530.

[0118] Vehicle 500 may include a GPU 520 (e.g., a separate GPU, or dGPU) which can be connected to SoC 504 via a high-speed interconnect (e.g., NVIDIA NVLINK). The GPU 520 can provide additional artificial intelligence capabilities, such as by running redundant and / or different neural networks, and may be used to train and / or update neural networks based on input from sensors in Vehicle 500 (e.g., sensor data).

[0119] Vehicle 500 may further include a network interface 524 which may include one or more wireless antennas 526 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas and Bluetooth® antennas). The network interface 524 may be used to enable wireless connectivity to a cloud over the Internet (e.g., with a server 578 and / or other network devices), to other vehicles, and / or to computing devices (e.g., passenger client devices). To communicate with other vehicles, a direct link may be established between two vehicles, and / or an indirect link may be established (e.g., over a network and over the Internet). The direct link may be provided using a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link can provide vehicle 500 information about vehicles in close proximity to vehicle 500 (e.g., vehicles in front of, beside, and / or behind vehicle 500). This functionality may also be part of the vehicle 500's joint adaptive cruise control functionality.

[0120] The network interface 524 may include an SoC that provides modulation and demodulation functions and enables the controller 536 to communicate over a wireless network. The network interface 524 may include a radio frequency front end for up-conversion from baseband to radio frequency and down-conversion from radio frequency to baseband. Frequency conversion can be performed through well-known processes and / or using a superheterodyne process. In some examples, the radio frequency front end functionality may be provided by a separate chip. The network interface may include wireless functionality for communication over LTE, WCDMA®, UMTS, GSM, CDMA2000, Bluetooth®, Bluetooth® LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0121] The vehicle 500 may further include a data store 528 which may include storage outside the chip (for example, outside the SoC 504). The data store 528 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash, hard disk, and / or other components and / or devices capable of storing at least one bit of data.

[0122] The vehicle 500 may further include GNSS sensors 558. The GNSS sensors 558 (e.g., GPS, assisted GPS sensors, differential GPS (DGPS) sensors, etc.) assist in mapping, perception, occupy grid generation, and / or route planning functions. Any number of GNSS sensors 558 may be used, including, but not limited to, GPS using a USB connector with Ethernet® to a serial (RS-232) bridge.

[0123] Vehicle 500 may further include a RADAR sensor 560. The RADAR sensor 560 may be used by vehicle 500 for long-range vehicle detection, even in darkness and / or severe weather conditions. The RADAR functional safety level may be ASIL B. In some examples, the RADAR sensor 560 may use CAN and / or bus 502 for control and to access object tracking data (for example, to transmit data generated by the RADAR sensor 560) using Ethernet® access for accessing raw data. A wide variety of RADAR sensor types may be used. For example, and without limitation, the RADAR sensor 560 may be suitable for front, rear, and side RADAR use. In some examples, a pulsed Doppler RADAR sensor may be used.

[0124] The RADAR sensor 560 may include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, and short-range side coverage. In some examples, the long-range RADAR may be used for adaptive cruise control functions. The long-range RADAR system can provide a wide field of view achieved by two or more independent scans, such as within a range of 250m. The RADAR sensor 560 can help distinguish between static and moving objects and may be used by ADAS systems for emergency brake assist and forward collision warning. The long-range RADAR sensor may include monostatic multimodal RADARs having multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In one example with six antennas, the four central antennas may create a focused beam pattern designed to record the area around the vehicle 500 at high speed with minimal interference from traffic in adjacent lanes. The other two antennas can widen the field of view, enabling rapid detection of vehicles entering or leaving the lane of the vehicle 500.

[0125] As an example, a medium-range RADAR system may include a range of up to 560m (front) or 80m (rear) and a field of view of up to 42 degrees (front) or 550 degrees (rear). A short-range RADAR system may include, but is not limited to, RADAR sensors designed to be mounted on both ends of the rear bumper. When mounted on both ends of the rear bumper, such a RADAR sensor system can create two beams that constantly monitor the blind spots behind and beside the vehicle.

[0126] Short-range radar systems can be used in ADAS systems for blind spot detection and / or lane change assistance.

[0127] The vehicle 500 may further include ultrasonic sensors 562. Positioned on the front, rear, and / or sides of the vehicle 500, the ultrasonic sensors 562 may be used for parking assistance and / or for creating and updating the occupancy grid. A wide variety of ultrasonic sensors 562 may be used, and different ultrasonic sensors 562 may be used for detection of different ranges (e.g., 2.5m, 4m). The ultrasonic sensors 562 may operate at a functional safety level of ASIL B.

[0128] The vehicle 500 may include a LiDAR sensor 564. The LiDAR sensor 564 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LiDAR sensor 564 may also have a functional safety level of ASIL B. In some examples, the vehicle 500 may include multiple LiDAR sensors 564 (e.g., two, four, six, etc.) that can use Ethernet® (for example, to provide data to a Gigabit Ethernet® switch).

[0129] In some examples, the LIDAR sensor 564 may have the ability to provide a list of objects and their distances within a 360-degree field of view. A commercially available LIDAR sensor 564 may have an advertised range of approximately 500m, for example, with an accuracy of 2cm to 3cm and support for 500Mbps Ethernet® connectivity. In some examples, one or more non-protruding LIDAR sensors 564 may be used. In such examples, the LIDAR sensor 564 may be implemented as a small device that can be incorporated into the front, rear, side, and / or corners of a vehicle 500. In such examples, the LIDAR sensor 564 may have a range of 200m even for low-reflection objects and can provide a field of view up to 120 degrees horizontal and 35 degrees vertical. A front-mounted LIDAR sensor 564 may be configured for a horizontal field of view between 45 and 135 degrees.

[0130] In some applications, LiDAR technologies such as 3D flash LiDAR may also be used. 3D flash LiDAR uses a laser flash as a source to illuminate the area around the vehicle up to approximately 200m. The flash LiDAR unit includes a receptor that records the laser pulse travel time and reflected light on each pixel, sequentially corresponding to the range from the vehicle to the object. Flash LiDAR can enable the generation of high-precision and distortion-free images of the surroundings with every laser flash. In some applications, four flash LiDAR sensors may be deployed, one on each side of the vehicle. Available 3D flash LiDAR systems include solid-state 3D steering array LiDAR cameras (e.g., non-scanning LiDAR devices) that have no moving parts other than a blower. The flash LiDAR device can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture reflected laser light in the form of a 3D range point cloud and co-documented intensity data. By using flash LiDAR, and because flash LiDAR is a solid-state device with no moving parts, the LiDAR sensor 564 may be less susceptible to motion blur, vibration, and / or shock.

[0131] The vehicle may further include an IMU sensor 566. In some examples, the IMU sensor 566 may be positioned in the center of the rear axle of the vehicle 500. The IMU sensor 566 may include, but is not limited to, an accelerometer, magnetometer, gyroscope, magnetic compass, and / or other sensor types. In some examples, such as in a 6-axis application, the IMU sensor 566 may include an accelerometer and a gyroscope, while in a 9-axis application, the IMU sensor 566 may include an accelerometer, a gyroscope, and a magnetometer.

[0132] In some embodiments, the IMU sensor 566 may be implemented as a miniature, high-performance GPS-aided inertial navigation system (GPS / INS) that combines a micro-electro-mechanical system (MEMS) inertial sensor, a high-sensitivity GPS receiver, and an advanced Kalman filtering algorithm to provide estimates of position, velocity, and attitude. As such, in some examples, the IMU sensor 566 may enable the vehicle 500 to estimate its direction of travel without requiring input from a magnetic sensor by directly observing and correlating changes in velocity from the GPS to the IMU sensor 566. In some embodiments, the IMU sensor 566 and the GNSS sensor 558 may be combined in a single integrated unit.

[0133] The vehicle may include a microphone 596 placed inside and / or around the vehicle 500. The microphone 596 may, among other things, be used for emergency vehicle detection and identification.

[0134] The vehicle may further include any number of camera types, including a stereo camera 568, a wide-view camera 570, an infrared camera 572, a surround camera 574, a long-range and / or medium-range camera 598, and / or other camera types. The cameras may be used to capture image data around the entire exterior surface of the vehicle 500. The type of camera used will depend on the embodiment and requirements of the vehicle 500, and any combination of camera types may be used to achieve the required coverage around the vehicle 500. In addition, the number of cameras may vary depending on the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. The cameras may, as an example, support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet®. Each camera is described in more detail herein in relation to Figures 5A and 5B.

[0135] The vehicle 500 may further include a vibration sensor 542. The vibration sensor 542 can measure vibrations of vehicle components, such as axles. For example, a change in vibration may indicate a change in the road surface. In another example, when two or more vibration sensors 542 are used, the difference in vibration may be used to determine friction or slippage of the road surface (for example, when the difference in vibration is between a power-driven axle and a free-rotating axle).

[0136] Vehicle 500 may include an ADAS system 538. In some examples, the ADAS system 538 may include a System of Control (SoC). The ADAS system 538 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward crash warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keep assist (LKA), blind spot warning (BSW), rear cross-traffic warning (RCTW), collision warning system (CWS), lane centering (LC), and / or other features and functions.

[0137] The ACC system may use a radar sensor 560, a lithium-ion sensor 564, and / or a camera. The ACC system may include longitudinal ACC and / or transverse ACC. Longitudinal ACC monitors and controls the distance to the vehicle immediately in front of vehicle 500 and automatically adjusts the vehicle speed to maintain a safe distance from the vehicle ahead. Transverse ACC performs distance maintenance and advises vehicle 500 to change lanes when necessary. Transverse ACC is related to other ADAS applications such as LCA and CWS.

[0138] CACC uses information from other vehicles that can be received from other vehicles via a wireless link through the network interface 524 and / or wireless antenna 526, or indirectly via a network connection (e.g., via the Internet). Direct links may be provided by vehicle-to-vehicle (V2V) communication links, while indirect links may be infrastructure-to-vehicle (I2V) communication links. Generally, the V2V communication concept provides information about the vehicle immediately ahead (e.g., a vehicle in the same lane as vehicle 500, immediately in front of vehicle 500), while the I2V communication concept provides information about traffic further ahead. A CACC system may include either or both I2V and V2V information sources. Given information about vehicles ahead of vehicle 500, CACC can be more reliable, and CACC has the potential to make traffic flow smoother and reduce road congestion.

[0139] The FCW system is designed to warn the driver of hazards so that the driver can take corrective action. The FCW system uses a forward-facing camera and / or radar sensor 560, coupled to a dedicated processor, DSP, FPGA, and / or ASIC, electrically coupled to driver feedback such as a display, speaker, and / or vibration components. The FCW system can provide warnings in the form of audible, visual, vibration, and / or quick brake pulses.

[0140] An AEB system can detect an imminent forward collision with another vehicle or object and automatically apply the brakes if the driver does not take corrective action within a specified time or distance parameter. The AEB system may use a forward-facing camera and / or radar sensor 560 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. When the AEB system detects a hazard, it typically first warns the driver to take corrective action to avoid the collision. If the driver does not take corrective action, the AEB system may automatically apply the brakes as part of an effort to prevent, or at least mitigate, the impact of the anticipated collision. The AEB system may include techniques such as dynamic brake support and / or impending collision braking.

[0141] The LDW system warns the driver when the vehicle 500 crosses a lane marking by providing visual, audible, and / or tactile warnings, such as vibration of the steering wheel or seat. The LDW system does not activate when the driver indicates an intentional lane departure by activating the turn signal. The LDW system may use a forward-facing camera connected to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback, such as a display, speaker, and / or vibration components.

[0142] The LKA system is a modified version of the LDW system. The LKA system provides steering input or braking to correct the vehicle 500 if it begins to drift out of its lane.

[0143] The BSW system detects and warns the driver of a vehicle in its blind spots. The BSW system can provide visual, audible, and / or tactile warnings to indicate that merging or changing lanes is unsafe. The system may provide additional warnings when the driver uses the turn signals. The BSW system may use a rear-facing camera and / or radar sensor 560, coupled to a dedicated processor, DSP, FPGA, and / or ASIC, electrically coupled to driver feedback, such as a display, speaker, and / or vibration component.

[0144] The RCTW system can provide visual, audible, and / or haptic notifications when an object is detected outside the range of the rear camera while the vehicle 500 is reversing. Some RCTW systems include AEB to ensure that the vehicle brakes are applied to avoid a collision. The RCTW system may use one or more rear-facing RADAR sensors 560, coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which are electrically coupled to driver feedback, such as a display, speaker, and / or vibration component.

[0145] Conventional ADAS systems warn the driver and allow the driver to determine whether a safe condition truly exists and act accordingly. However, conventional ADAS systems have sometimes tended to produce misjudgments that, while not usually catastrophic, can be troubling and distracting to the driver. In the autonomous vehicle 500, however, if the results are contradictory, the vehicle 500 itself must decide whether to heed the results from the primary computer or the secondary computer (e.g., the first controller 536 or the second controller 536). For example, in some embodiments, the ADAS system 538 may also be a backup and / or secondary computer for providing perceptual information to a backup computer rationality module. The backup computer rationality monitor can run a variety of redundant software on hardware components to detect failures in perceptual and dynamic driving tasks. The output from the ADAS system 538 may be provided to the supervisory MCU. If the outputs from the primary and secondary computers are contradictory, the supervisory MCU must decide how to reconcile the contradiction to ensure safe operation.

[0146] In some implementations, a primary computer may be configured to provide a supervising MCU with a reliability score indicating the reliability of the primary computer in a selected outcome. If the reliability score exceeds a threshold, the supervising MCU may follow the primary computer's instructions, regardless of whether the secondary computer gives conflicting or inconsistent results. If the reliability score does not meet the threshold, and the primary and secondary computers produce different results (e.g., conflicting results), the supervising MCU may mediate between the computers to determine an appropriate outcome.

[0147] The supervisory MCU may be configured to run a neural network trained and configured to determine, based on the outputs from the primary and secondary computers, when a secondary computer is providing a false alarm. Thus, the neural network in the supervisory MCU can learn when the output of the secondary computer is reliable and when it is not. For example, when the secondary computer is a radar-based forward crossing (FCW) system, the neural network in the supervisory MCU can learn when the FCW is identifying metal objects that are not actually dangerous, such as sewer grates or manhole covers that trigger an alarm. Similarly, when the secondary computer is a camera-based lane departure warning (LDW) system, the neural network in the supervisory MCU can learn to ignore the LDW when a cyclist or pedestrian is present and lane departure is actually the safest operation. In embodiments involving a neural network running on the supervisory MCU, the supervisory MCU may include at least one of a DLA or GPU suitable for running a neural network with associated memory. In a preferred embodiment, the supervisory MCU may comprise and / or be included as a component of the SoC504.

[0148] In other examples, ADAS system 538 may include a secondary computer that performs ADAS functions using conventional rules of computer vision. As such, the secondary computer may use classical computer vision rules (if-then), and the presence of a neural network within the supervisory MCU can improve reliability, safety, and performance. For example, diverse implementations and intentional non-identities make the entire system more fault-tolerant, particularly to failures caused by software (or software-hardware interface) functions. For instance, if a software bug or error exists in the software running on the primary computer, and non-identical software code running on the secondary computer produces the same overall result, the supervisory MCU may have greater confidence that the overall result is correct and that the bug in the software or hardware on the primary computer did not cause a critical error.

[0149] In some examples, the output of the ADAS system 538 may be supplied to the perception block and / or the dynamic driving task block of the primary computer. For example, if the ADAS system 538 indicates a forward collision warning due to an object immediately ahead, the perception block can use this information when identifying the object. In other examples, the secondary computer may have its own neural network, which is trained as described herein and therefore reduces the risk of misjudgment.

[0150] Vehicle 500 may further include an infotainment SoC 530 (for example, an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system does not have to be an SoC and may include two or more separate components. The infotainment SoC 530 may include a combination of hardware and software that can be used to provide vehicle 500 with audio (e.g., music, personal digital assistant, navigation commands, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and / or information services (e.g., navigation system, rear parking assist, radio data system, fuel level, total mileage, brake fuel level, oil level, door open / close, air filter information, and other vehicle-related information). For example, the infotainment SoC 530 may also include wireless, disc player, navigation system, video player, USB and Bluetooth® connectivity, car computer, in-car entertainment, Wi-Fi, steering wheel audio control unit, hands-free voice control, heads-up display (HUD), HMI display 534, telematics device, control panel (for example, for controlling and / or interacting with various components, features, and / or systems), and / or other components. The infotainment SoC 530 may be further used to provide information (for example, visual and / or audible) to the vehicle user, such as information from the ADAS system 538, autonomous driving information such as planned vehicle operation, trajectory, surrounding environment information (for example, intersection information, vehicle information, road information, etc.), and / or other information.

[0151] The infotainment SoC 530 may include GPU functionality. The infotainment SoC 530 can communicate with other devices, systems, and / or components of the vehicle 500 via bus 502 (e.g., CAN bus, Ethernet®, etc.). In some examples, the infotainment SoC 530 may be coupled to a supervisory MCU so that the infotainment system's GPU can perform certain self-drive functions in the event of a primary controller 536 (e.g., the vehicle 500's primary and / or backup computer) failure. In such examples, the infotainment SoC 530 can put the vehicle 500 into a chauffeur-safe stop mode as described herein.

[0152] Vehicle 500 may further include an instrument cluster 532 (e.g., a digital dash, an electronic instrument cluster, a digital instrument panel, etc.). The instrument cluster 532 may include a controller and / or a supercomputer (e.g., a separate controller or supercomputer). The instrument cluster 532 may include a set of instruments such as a speedometer, fuel level indicator, oil pressure indicator, tachometer, odometer, turn signals, gear shift position indicator, seat belt warning light, parking brake warning light, engine fault light, airbag (SRS) system information, lighting control device, safety system control device, and navigation information. In some examples, information may be displayed and / or shared between the infotainment SoC 530 and the instrument cluster 532. In other words, the instrument cluster 532 may be included as part of the infotainment SoC 530, and vice versa.

[0153] Figure 5D is a system diagram of communication between the cloud-based server of Figure 5A and an exemplary autonomous vehicle 500, according to some embodiments of the present disclosure. System 576 may include a server 578, a network 590, and a vehicle including the vehicle 500. Server 578 may include a plurality of GPUs 584(A) to 584(H) (collectively referred to herein as GPU 584), PCIe switches 582(A) to 582(H) (collectively referred to herein as PCIe switch 582), and / or CPUs 580(A) to 580(B) (collectively referred to herein as CPU 580). The GPUs 584, CPUs 580, and PCIe switches may be interconnected by high-speed interconnects, such as, for example, NVLink interfaces 588 and / or PCIe connections 586 developed by NVIDIA. In some examples, the GPU584 is connected via NVLink and / or NVSwitch SoCs, and the GPU584 and PCIe switch 582 are connected via PCIe interconnects. Eight GPU584s, two CPU580s, and two PCIe switches are illustrated, but this is not intended to be an limitation. Depending on the embodiment, each server 578 may contain any number of GPU584s, CPU580s, and / or PCIe switches. For example, each server 578 may contain eight, sixteen, thirty-two, and / or more GPU584s.

[0154] Server 578 can receive image data from vehicles via network 590, representing images showing unexpected or altered road conditions, such as recently started road construction. Server 578 can transmit map information 594, including information about traffic and road conditions, to vehicles via network 590, including information about the neural network 592, updated neural network 592, and / or map information 594. Updates to map information 594 may include updates to HD map 522, such as information about construction sites, potholes, detours, floods, and / or other obstacles. In some instances, the neural network 592, updated neural network 592, and / or map information 594 may have arisen from new training and / or experience represented in data received from any number of vehicles in the environment, and / or based on training performed in a data center (for example, using server 578 and / or other servers).

[0155] Server 578 may be used to train a machine learning model (e.g., a neural network) based on training data. The training data may be generated by a vehicle and / or in a simulation (e.g., using a game engine). In some instances, the training data is tagged (e.g., if the neural network benefits from supervised learning) and / or otherwise pre-processed, while in other instances, the training data is not tagged and / or pre-processed (e.g., if the neural network does not require supervised learning). Training may be performed according to any one or more classes of machine learning techniques, including but not limited to the following: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, associative learning, transfer learning, feature learning (including key component and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including pre-dictionary learning), rule-based machine learning, anomaly detection, and variations or combinations thereof. After the machine learning model has been traced, the machine learning model may be used by the vehicle (for example, transmitted to the vehicle via network 590), and / or the machine learning model may be used by server 578 to remotely monitor the vehicle.

[0156] In some examples, Server 578 can receive data from vehicles and apply it to a state-of-the-art real-time neural network for real-time intelligent inference. Server 578 may include deep learning supercomputers and / or dedicated AI computers powered by GPU 584, such as the DGX and DGX Station Machines developed by NVIDIA. However, in some examples, Server 578 may include deep learning infrastructure that uses only CPU-powered data centers.

[0157] The deep learning infrastructure of server 578 can have the capability for high-speed real-time inference, which can be used to evaluate and verify the condition of the processor, software, and / or associated hardware within vehicle 500. For example, the deep learning infrastructure can receive periodic updates from vehicle 500, such as images of a sequence and / or objects located within images of that sequence (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure can run its own neural network to identify objects and compare them with objects identified by vehicle 500, and if the results do not match and the infrastructure concludes that the AI ​​within vehicle 500 is not functioning properly, server 578 can send a signal to vehicle 500 instructing the vehicle's fail-safe computer to infer control, notify passengers, and complete a safe parking operation.

[0158] For inference, server 578 may include GPU 584 and one or more programmable inference accelerators (e.g., NVIDIA TensorRT). The combination of a GPU-powered server and inference accelerator can enable real-time responsiveness. In other examples, such as when high performance is not required, a server powered by a CPU, FPGA, and other processors may be used for inference.

[0159] Exemplary computing devices Figure 6 is a block diagram of an example of a computing device 600 suitable for use in implementing some embodiments of the present disclosure. The computing device 600 may include an interconnection system 602 that indirectly or directly connects the following devices: memory 604, one or more central processing units (CPUs) 606, one or more graphics processing units (GPUs) 608, a communication interface 610, input / output (I / O) ports 612, input / output components 614, a power supply device 616, one or more presentation components 618 (e.g., a display), and one or more logical units 620. In at least one embodiment, the computing device 600 may include one or more virtual machines (VMs), and / or any of its components may include virtual components (e.g., virtual hardware components). As an unrestricted example, one or more of the GPUs 608 may include one or more vGPUs, one or more of the CPUs 606 may include one or more vCPUs, and / or one or more of the logical units 620 may include one or more virtual logical units. As such, the computing device 600 may include individual components (e.g., an entire GPU dedicated to the computing device 600), virtual components (e.g., a portion of a GPU dedicated to the computing device 600), or a combination thereof.

[0160] The various blocks in Figure 6 are shown connected by lines via the interconnection system 602, but this is not intended to be restrictive and is simply for clarity. For example, in some embodiments, a presentation component 618, such as a display device, could be considered an I / O component 614 (for example, if the display is a touchscreen). As another example, the CPU 606 and / or GPU 608 may include memory (for example, memory 604 may represent a storage device in addition to the memory of the GPU 608, CPU 606, and / or other components). In other words, the computing devices in Figure 6 are merely illustrative. Categories such as “workstation,” “server,” “laptop,” “desktop,” “tablet,” “client device,” “mobile device,” “handheld device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types are all intended to fall within the scope of the computing devices in Figure 6 and are therefore not distinguished.

[0161] The interconnection system 602 may represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnection system 602 may include one or more bus or link types, such as an industry standard architecture (ISA) bus, an extended industry standard architecture (EISA) bus, a VESA (video electronics standards association) bus, a peripheral component interconnect (PCI) bus, a peripheral component interconnect express (PCIe) bus, and / or other types of buses or links. In some embodiments, direct connections exist between components. For example, the CPU 606 may be directly connected to the memory 604. Furthermore, the CPU 606 may be directly connected to the GPU 608. Where direct or point-to-point connections exist between components, the interconnection system 602 may include PCIe links to implement the connections. In these examples, the PCI bus does not need to be included in the computing device 600.

[0162] Memory 604 may include any of various computer-readable media. The computer-readable media may be any available media accessible by the computing device 600. The computer-readable media may include both volatile and non-volatile media, and removable and non-removable media. For example, but not limited to, the computer-readable media may include computer storage media and communication media.

[0163] Computer storage media may include both volatile and non-volatile media and / or removable and non-removable media implemented in any method or technique for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 604 may store computer-readable instructions (e.g., representing programs and / or program elements), such as an operating system. Computer storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory, or other memory technologies, CD-ROM, digital versatile disk (DVD), or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage, or other magnetic storage devices, or any other media that can be used to store desired information and can be accessed by computing device 600. In this specification, computer storage media does not include signals themselves.

[0164] Computer storage media include any information distribution medium that can implement computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals such as carrier waves or other transfer mechanisms. The term “modulated data signal” may refer to a signal that has been modified in a manner that has one or more of its characteristic sets or encodes information within the signal. For example, but not limited to, computer storage media may include wired media such as wired networks or direct wired connections, and wireless media such as acoustic, RF, infrared, and other wireless media. Any combination of the foregoing should also be included in the scope of computer-readable media.

[0165] The CPU 606 may be configured to execute at least some computer-readable instructions to control one or more components of the computing device 600 to execute one or more of the methods and / or processes described herein. The CPU 606 may include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) each capable of processing a large number of software threads simultaneously. The CPU 606 may include any type of processor, and depending on the type of computing device 600 in which it is implemented, it may include different types of processors (e.g., a processor with fewer cores for mobile devices and a processor with more cores for servers). For example, depending on the type of computing device 600, the processor may be an Advanced RISC Machines (ARM) processor implemented using Reduced Instruction Set Computing (RISC), or an x86 processor implemented using Complex Instruction Set Computing (CISC). The computing device 600 may include one or more CPUs 606 in one or more microprocessors or auxiliary coprocessors, such as a computing coprocessor.

[0166] In addition to or instead of the CPU 606, the GPU 608 may be configured to execute at least some computer-readable instructions to control one or more components of the computing device 600 to execute one or more of the methods and / or processes described herein. One or more of the GPU 608 may be an integrated GPU (for example, with one or more of the CPU 606, and / or one or more of the GPU 608 may be a discrete GPU. In embodiments, one or more of the GPU 608 may be a coprocessor of one or more of the CPU 606. The GPU 608 may be used by the computing device 600 to render graphics (for example, 3D graphics) or to perform general-purpose computing. For example, the GPU 608 may be used for GPU-based general-purpose computing (GPGPU). It may be used for a GPU. The GPU608 may include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. The GPU608 can generate pixel data for an output image in response to rendering commands (for example, rendering commands from CPU606 received via the host interface). The GPU608 may include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. Display memory may be included as part of memory 604. A GPU608 may include two or more GPUs operating in parallel (for example, via a link). The link can connect directly to the GPUs (for example, using NVLINK) or via a switch (for example, using NVSwitch). When coupled together, each GPU608 can generate pixel data or GPGPU data for different parts of an output or different outputs (for example, the first GPU for the first image and the second GPU for the second image). Each GPU may have its own memory or may share memory with other GPUs.

[0167] In addition to or instead of the CPU 606 and / or GPU 608, the logic unit 620 may be configured to execute at least some computer-readable instructions to control one or more of the computing devices 600 to execute one or more of the methods and / or processes described herein. In embodiments, the CPU 606, GPU 608, and / or logic unit 620 can execute any combination of methods, processes, and / or parts thereof discretely or congruently. One or more of the logic units 620 may be part of and / or integrated with one or more of the CPU 606 and / or GPU 608, and / or one or more of the logic units 620 may be discrete components of the CPU 606 and / or GPU 608 or otherwise external to them. In embodiments, one or more of the logic units 620 may be coprocessors of one or more of the CPU 606 and / or one or more of the GPU 608.

[0168] Examples of logical unit 620 include one or more processing cores and / or components thereof, such as a Data Processing Unit (DPU), Tensor Core (TC), Tensor Processing Unit (TPU), Pixel Visual Core (PVC), Vision Processing Unit (VPU), Graphics Processing Cluster (GPC), Texture Processing Cluster (TPC), Streaming Multiprocessor (SM), Tree Traversal Unit (TTU), Artificial Intelligence Accelerator (AIA), and Deep Learning Accelerator (DLA). This includes an Accelerator, a Logical Unit (ALU), an Application-Specific Integrated Circuit (ASIC), a Floating-Point Unit (FPU), input / output (I / O) elements, a Peripheral Component Interconnect (PCI) or Peripheral Component Interconnect Express (PCIe) element, and / or similar.

[0169] The communication interface 610 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 600 to communicate with other computing devices via an electronic communication network, including wired and / or wireless communication. The communication interface 610 may include components and functions to enable communication over any of several different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth®, Bluetooth® LE, ZigBee, etc.), wired networks (e.g., communicating via Ethernet® or InfiniBand), low-power wide-area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, the logic unit 620 and / or the communication interface 610 may include one or more data processing units (DPUs) to directly transmit data received via the network and / or via the interconnection system 602 to one or more GPUs 608 (e.g., their memory).

[0170] I / O port 612 can enable the computing device 600 to be logically connected to other devices, including I / O components 614, presentation components 618, and / or other components, some of which can be built into (e.g., integrated into) the computing device 600. Exemplary I / O components 614 include microphones, mice, keyboards, joysticks, gamepads, game controllers, satellite dishes, scanners, printers, wireless devices, etc. I / O components 614 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by the user. In some cases, the input may be transmitted to appropriate network elements for further processing. The NUI may implement any combination of voice recognition, stylus recognition, face recognition, biometric recognition, on-screen and beside-screen gesture recognition, air gestures, head and target tracking, and touch recognition related to the display of the computing device 600 (as described in more detail below). The computing device 600 may include depth cameras, such as stereoscope camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof, for gesture detection and recognition. Additionally, the computing device 600 may include accelerometers or gyroscopes that enable motion detection (for example, as part of an inertia measurement unit (IMU)). In some examples, the output of the accelerometer or gyroscope may be used by the computing device 600 to render immersive augmented reality or virtual reality.

[0171] The power supply device 616 may include a hardwired power supply device, a battery power supply device, or a combination thereof. The power supply device 616 can provide power to the computing device 600 to enable the components of the computing device 600 to operate.

[0172] The presentation component 618 may include a display (e.g., a monitor, touch screen, television screen, head-up display device (HUD), other display types, or a combination thereof), a speaker, and / or other presentation components. The presentation component 618 can receive data from other components (e.g., GPU 608, CPU 606, DPU, etc.) and output data (e.g., as images, videos, sounds, etc.).

[0173] Exemplary data center Figure 7 shows an exemplary data center 700 that may be used in at least one embodiment of the present disclosure. The data center 700 may include a data center infrastructure layer 710, a framework layer 720, a software layer 730, and / or an application layer 740.

[0174] As shown in Figure 7, the data center infrastructure layer 710 may include a resource orchestrator 712, grouped computing resources 714, and node computing resources ("node CRs") 716(1) to 716(N), where "N" represents any integer or natural number. In at least one embodiment, the node CRs 716(1) to 716(N) may include, but are not limited to, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field-programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output (NW I / O) devices, network switches, virtual machines (VMs), power modules, and / or cooling modules. In some embodiments, one or more nodes CR716(1) to 716(N) may correspond to a server having one or more of the aforementioned computing resources. In addition, in some embodiments, nodes CR716(1) to 7161(N) may include one or more virtual components, such as vGPUs, vCPUs, and / or similar, and / or one or more nodes CR716(1) to 716(N) may correspond to a virtual machine (VM).

[0175] In at least one embodiment, the grouped computing resources 714 may include a separate group of nodes CR716 housed in one or more racks (not shown), or a number of racks housed in data centers in various geographical locations (also not shown). The separate group of nodes CR716 within the grouped computing resources 714 may include grouped compute, network, memory, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, several nodes CR716, including CPUs, GPUs, DPUs, and / or other processors, may be grouped in one or more racks to provide computing resources to support one or more workloads. The one or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0176] The resource orchestrator 712 can configure or otherwise control one or more nodes CR716(1) to 716(N) and / or grouped computing resources 714. In at least one embodiment, the resource orchestrator 712 may include a software design infrastructure (SDI) management entity of the data center 700. The resource orchestrator 712 may include hardware, software, or any combination thereof.

[0177] In at least one embodiment, as shown in Figure 7, the framework layer 720 may include a job scheduler 732, a configuration manager 734, a resource manager 736, and / or a distributed file system 738. The framework layer 720 may include a framework to support software 732 of the software layer 730 and / or one or more applications 742 of the application layer 740. The software 732 or application 742 may include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure, respectively. The framework layer 720 may be, but is not limited to, a type of free and open-source software web application framework, such as Apache Spark® ("Spark"), which may use the distributed file system 738 for large-scale data processing (e.g., "big data"). In at least one embodiment, the job scheduler 732 may include a Spark driver to facilitate scheduling of workloads supported by various layers of the data center 700. The configuration manager 734 may have the ability to configure different layers, for example, a software layer 730 and a framework layer 720 including Spark and a distributed file system 738 to support large-scale data processing. The resource manager 736 may have the ability to manage clustered or grouped computing resources mapped or allocated for support of the distributed file system 738 and the job scheduler 732. In at least one embodiment, the clustered or grouped computing resources may include computing resources 714 grouped in the data center infrastructure layer 710. The resource manager 736 can coordinate with the resource orchestrator 712 to manage these mapped or allocated computing resources.

[0178] In at least one embodiment, the software 732 included in the software layer 730 may include software used by at least a portion of nodes CR716(1) to 716(N), grouped computing resources 714, and / or the distributed file system 738 of the framework layer 720. One or more types of software may include, but are not limited to, internet web page search software, email virus scanning software, database software, and streaming video content software.

[0179] In at least one embodiment, the application 742 included in the application layer 740 may include one or more types of applications used by at least a portion of the nodes CR716(1) to 716(N), the grouped computing resources 714, and / or the distributed file system 738 of the framework layer 720. One or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments.

[0180] In at least one embodiment, any of the configuration manager 734, resource manager 736, and resource orchestrator 712 may implement any number and type of self-rewriting actions based on any amount and type of data obtained in any technically possible manner. Self-rewriting actions may free the data center operator of data center 700 from making potentially poor configuration decisions and possibly avoiding underutilized and / or underperforming parts of the data center.

[0181] The data center 700 may include tools, services, software, or other resources for training one or more machine learning models or for predicting or inferring information using one or more machine learning models according to one or more embodiments described herein. For example, a machine learning model may be trained by calculating weight parameters by a neural network architecture using the software and / or computing resources described herein with respect to the data center 700. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks may be used to infer or predict information using the resources described herein with respect to the data center 700 by using weight parameters calculated via one or more training techniques, not limited to those described herein.

[0182] In at least one embodiment, the data center 700 may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, and / or other hardware (or corresponding virtual computing resources) for performing training and / or inference using the aforementioned resources. Furthermore, one or more of the aforementioned software and / or hardware resources may be configured as services that enable users to train or perform inference of information, such as image recognition, speech recognition, or other artificial intelligence services.

[0183] Exemplary network environment A network environment suitable for use in implementing the embodiments of this disclosure may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. Each client device, server, and / or other device type (e.g., each device) may be implemented as one or more instances of the computing device 600 in Figure 6, for example, each device may include similar components, features, and / or functionalities of the computing device 600. In addition, if backend devices (e.g., servers, NAS, etc.) are implemented, they may be included as part of the data center 700, examples of which are further detailed herein with respect to Figure 7.

[0184] Components of a network environment may communicate with one another via the network, either wired, wirelessly, or both. A network may include multiple networks, or a network of networks. For example, a network may include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks, such as the Internet and / or the Public Switched Telephone Network (PSTN), and / or one or more private networks. If a network includes a wireless telecommunications network, its components, such as base stations, towers, or access points (and other components), may provide wireless connectivity.

[0185] Compatible network environments may include one or more peer-to-peer network environments (in which case servers may not be included in the network environment) and one or more client-server network environments (in which case one or more servers may be included in the network environment). In a peer-to-peer network environment, the functionality described herein with respect to the server can be implemented on any number of client devices.

[0186] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, or a combination thereof. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more of the servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework to support the software in the software layer and / or one or more applications in the application layer. The software or applications may each include web-based service software or applications. In the embodiment, one or more of the client devices may use the web-based service software or applications (for example, by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a type of free and open-source software web application framework that may use a distributed file system for, for example, large-scale data processing (e.g., “big data”).

[0187] A cloud-based network environment may provide cloud computing and / or cloud storage that implements any combination of the computing and / or data storage functions (or one or more of them) described herein. Any of these various functions may be distributed across multiple locations from a central or core server (e.g., one or more data centers that may be distributed across states, territories, countries, or the world). If the connection to the user (e.g., a client device) is relatively close to the edge server, the core server may delegate at least a portion of its functionality to the edge server. The cloud-based network environment may be private (e.g., limited to a single organization), public (e.g., available to multiple organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0188] A client device may include at least some of the components, features, and functionalities of the exemplary computing device 600 described herein with respect to Figure 6. As an example, and not limited to, a client device may be implemented as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, video camera, surveillance device or system, vehicle, boat, airship, virtual machine, drone, robot, handheld communication device, hospital device, gaming device or system, entertainment system, vehicle computer system, embedded system controller, remote control, instrument, consumer electronic device, workstation, edge device, any combination of these depicted devices, or any other suitable device.

[0189] This disclosure may be described in general terms with computer code or machine-usable instructions, including computer-executable instructions such as program modules, which are executed by computers or other machines, such as personal digital assistants or other handheld devices. Generally, a program module, including routines, programs, objects, components, and data structures, refers to code that performs a specific task or implements a specific abstract data type. This disclosure may be implemented in a variety of configurations, including handheld devices, consumer electronics, general-purpose computers, and more specialized computing devices. This disclosure may also be implemented in a distributed computing environment where tasks are performed by remote processing devices linked over a communication network.

[0190] In this specification, any “and / or” statement relating to two or more elements should be interpreted as meaning only one element or a combination of elements. For example, “element A, element B, and / or element C” may include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. In addition, “at least one of element A or element B” may include at least one element A, at least one element B, or at least one element A and at least one element B. Furthermore, “at least one of element A and element B” may include at least one element A, at least one element B, or at least one element A and at least one element B.

[0191] The subject matter of this disclosure is described in a manner that is specific in order to satisfy statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors intend that the claimed subject matter may be carried out in other ways, including different steps or combinations of steps similar to those described herein, in conjunction with other current or future technologies. Furthermore, the terms “step” and / or “block” may be used herein to imply different elements of the way in which they are used, but these terms should not be construed as implying any particular order among the various steps disclosed herein unless the order of the individual steps is expressly stated and, when so, expressed.

Claims

1. Processing circuit A processor comprising, the processing circuit, Create a first tracking state for a first object representing a first probability distribution and a second tracking state for a second object representing a second probability distribution. The current obstacle boundary line corresponding to the current detection of one of the first or second objects is determined at least in part based on LiDAR measurements. The current obstacle boundary line is compared with a first obstacle boundary line corresponding to the first tracking state in order to generate a first velocity estimate, and a second obstacle boundary line corresponding to the second tracking state in order to generate a second velocity estimate. A first cost function using the first probability distribution and the first velocity estimate, and a second cost function using the second probability distribution and the second velocity estimate are calculated. Based at least in part on the fact that the first cost function is smaller than the second cost function, it is determined that the current obstacle boundary line corresponds most closely to the first obstacle boundary line. A processor that updates the first tracking state using the first velocity estimate to generate an updated first tracking state corresponding to the first object.

2. The processor according to claim 1, further comprising a processing circuit for converting the first tracking state and the second tracking state into a coordinate system corresponding to the current detection to explain the ego motion of the ego machine.

3. The first tracking state corresponds to a first Kalman filter with respect to the first velocity vector of the first obstacle boundary line, The second tracking state corresponds to a second Kalman filter with respect to the second velocity vector of the second obstacle boundary line, The processor according to claim 1.

4. The first obstacle boundary line includes a first shape metric that represents a first geometric shape, The current obstacle boundary line includes a second shape metric that represents a second geometric shape, The determination that the current obstacle boundary line most closely corresponds to the first obstacle boundary line is further based at least partially on the first geometric shape and the second geometric shape. The processor according to claim 1.

5. The processor according to claim 1, wherein the updated first tracking state has lower uncertainty than the first tracking state.

6. The processor according to claim 1, further comprising a processing circuit for filtering out one or more tracking states other than the first tracking state and the second tracking state by executing a gating function using the Mahalanobis distance.

7. The aforementioned processor, Control systems for autonomous or semi-autonomous machines, Perceptual systems for autonomous or semi-autonomous machines, A system for performing simulation operations. A system for performing deep learning operations. Systems implemented using edge devices, Systems implemented using robots, A system that incorporates one or more virtual machines (VMs). A system that is at least partially implemented in a data center, or A system that is at least partially implemented using cloud computing resources. The processor according to claim 1, which is included in at least one of the following.

8. A system comprising a processing circuit, The processing circuit described above Create a first tracking state for a first object representing a first probability distribution and a second tracking state for a second object representing a second probability distribution. The current obstacle boundary line corresponding to the current detection of one of the first or second objects is determined at least in part based on LiDAR measurements. The current obstacle boundary line is compared with a first obstacle boundary line corresponding to the first tracking state in order to generate a first velocity estimate, and a second obstacle boundary line corresponding to the second tracking state in order to generate a second velocity estimate. A first cost function using the first probability distribution and the first velocity estimate, and a second cost function using the second probability distribution and the second velocity estimate are calculated. Based at least in part on the fact that the first cost function is smaller than the second cost function, it is determined that the current obstacle boundary line corresponds most closely to the first obstacle boundary line. A system that updates the first tracking state using the first velocity estimate to generate an updated first tracking state corresponding to the first object.

9. A method performed by a processing circuit, Create a first tracking state for a first object representing a first probability distribution and a second tracking state for a second object representing a second probability distribution. The current obstacle boundary line corresponding to the current detection of one of the first or second objects is determined at least in part based on LiDAR measurements. The current obstacle boundary line is compared with a first obstacle boundary line corresponding to the first tracking state in order to generate a first velocity estimate, and a second obstacle boundary line corresponding to the second tracking state in order to generate a second velocity estimate. A first cost function using the first probability distribution and the first velocity estimate, and a second cost function using the second probability distribution and the second velocity estimate are calculated. Based at least in part on the fact that the first cost function is smaller than the second cost function, it is determined that the current obstacle boundary line corresponds most closely to the first obstacle boundary line. A method for updating the first tracking state using the first velocity estimate to generate an updated first tracking state corresponding to the first object.

Images