Autonomous and semi-autonomous systems and hazard detection in applications

Abstract

Embodiments relate to autonomous and semi-autonomous systems and hazard detection in applications. The converter may use the sampled images and LiDAR features to extract and decode a representation of the presence of a hazard, the shape of the hazard, and / or the class thereof at a 3D location corresponding to each initial converter query. These detections may be provided to one or more control components of an autonomous vehicle that may use these detections to navigate, plan, or otherwise perform one or more operations (e.g., obstacle avoidance, lane keeping, lane changing, merging, lane splitting, etc. Some embodiments employ an automated approach to derive truth-valued data from sensor data collected by a data collection vehicle, such as data representing detected static scene points, navigable spatial boundaries, or detected hazardous objects. Thus, dangers such as road debris and other obstacles may be detected, and truth data may be generated for various sensing tasks.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 631,449, filed April 8, 2024, the contents of which are hereby incorporated by reference in their entirety. Background Art

[0003] Designing systems that can autonomously, safely, and comfortably drive a vehicle without supervision is extremely difficult. An autonomous vehicle should be able to navigate the vehicle's path through the surrounding three-dimensional (3D) environment, at least functionally equivalent to a dedicated driver—one that utilizes a perception and action system with an impressive ability to identify and react to dynamic and static hazards in complex environments. Therefore, the ability to detect hazards and road profiles is often crucial for autonomous driving perception systems. For example, estimated road profiles can be used for many important tasks, such as estimating the navigable space (e.g., the road surface), facilitating the detection of static obstacles on the road surface, adjusting the suspension or other components for a smoother ride, and / or estimating the height of static obstacles. Hazard detection may attempt to identify potential threats, such as dynamic obstacles (e.g., other vehicles, pedestrians, animals) or static obstacles (e.g., road debris, road construction barriers, road signs, traffic cones, curbs, guardrails, etc.). These technologies may rely on sensor data from cameras, LiDAR, radar, or ultrasonic sensors to provide a comprehensive and real-time understanding of the vehicle's surroundings.

[0004] Detecting distant hazards and road curves is particularly challenging due to the inherent limitations of current sensor technology. LiDAR uses laser pulses to create a detailed 3D representation of the environment, typically providing good accuracy at close range but producing sparse data points at longer distances. This can be further limited by weather conditions (for example, resulting in few or no measurements on slippery roads), leading to incomplete and less reliable representations. RADAR, on the other hand, uses radio waves to detect objects and measure distances. It can operate effectively at long ranges and in various weather conditions, but its lower resolution makes it difficult to accurately identify and classify distant hazards. Camera-only solutions provide high-resolution visual data, capturing a detailed image of the surrounding environment. However, these solutions struggle to accurately determine the distance to objects, especially those at a distance. Camera-based depth perception relies on visual cues and complex algorithms, which can be unreliable and / or computationally intensive. Furthermore, cameras are often sensitive to lighting conditions, such as glare from the sun or poor visibility in low light conditions, further complicating the task of detecting and assessing distant hazards.

[0005] Therefore, there is a need for improved hazard detection and surface sensing technologies. Summary of the Invention

[0006] Embodiments of the present disclosure relate to hazard detection and surface sensing in autonomous and semi-autonomous systems and applications. Disclosed are systems and methods for detecting and navigating (e.g., road) surfaces, detecting and avoiding hazards, and / or generating corresponding ground truth data for various detection networks or other machine learning models, such as those used by autonomous vehicles, semi-autonomous vehicles, robots, and / or other types of objects or machines.

[0007] Taking hazard detection as an example, a transformer can use sampled image and LiDAR features to extract and decode the presence, shape, and / or class of a hazard at a 3D location corresponding to each initial transformer query. For each initial query, the output layer of the transformer can regress a representation of a two-dimensional (2D) or 3D bounding box (or other bounding shape) anchored at the corresponding 3D location and predicted to contain the detected hazard, can regress a representation of uncertainty in the regressed bounding shape, and / or can classify the detected hazard into any number of supported classes (e.g., generating a corresponding class confidence score, such as a binary classification score, indicating whether the predicted road debris is present at the corresponding 3D location). These detection results can be provided to one or more control components of the autonomous vehicle, which can use these detection results to navigate, plan, or otherwise perform one or more operations (e.g., obstacle avoidance, lane keeping, lane changing, merging, lane splitting, etc.).

[0008] Additionally or alternatively, a transformer can be used to generate a representation of one or more features of a specified portion of a (e.g., a road or other navigable) surface. For example, a desired surface (e.g., a road) can be modeled using a set of sampled 3D points (e.g., along one or more 2D trajectories of an ego-machine), and the transformer can use the sampled image and LiDAR features to extract and decode representations of one or more features of each point (e.g., refined height, range, driving condition, etc.). For example, the output layer of the transformer can regress a representation of the refined height value at the 3D location corresponding to each initial transformer query, can regress a representation of the uncertainty in the regressed height value, can regress a representation of the driving condition at the point (e.g., quantifying surface damage caused by detected surface or weather conditions), and / or can classify the point into any number of supported classes (e.g., generating corresponding class confidence scores). These detection results can be provided to one or more control components of the autonomous vehicle, which can use these detection results to navigate, plan, or otherwise perform one or more maneuvers in the environment (e.g., avoiding obstacles or bumps, lane keeping, lane changing, merging, lane splitting, adjusting the suspension system of the ego object or ego actor to match the current road surface, applying advance acceleration or deceleration based on an approaching road surface slope, mapping, etc.).

[0009] In some embodiments, ground truth data for training neural networks and / or parameter tuning of classical machine learning models for detecting objects and / or surface features may be generated in various ways. For example, some embodiments may employ automated methods (e.g., using classical non-machine learning algorithms) to derive various types of ground truth data from sensor data collected by one or more data collection vehicles, such as data representing detected dynamic obstacles, detected ground models, detected surface features, detected static scene points, detected navigable space boundaries, detected hazardous objects, detected non-static scene points, detected weather and / or surface condition labels, and / or detected per-point artifact labels.

[0010] Thus, the techniques described herein can be used to detect hazards such as road debris and other obstacles, detect surface features such as ground elevation along a predicted trajectory, and / or generate ground truth data for various autonomous vehicle and / or other sensing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The present systems and methods for hazard detection and surface sensing in autonomous and semi-autonomous systems and applications are described in detail below with reference to the accompanying drawings, in which:

[0012] Figure 1is a data flow diagram illustrating an example object detection pipeline according to some embodiments of the present disclosure;

[0013] Figure 2 shows an example elevation curve of a road surface according to some embodiments of the present disclosure;

[0014] Figure 3 is a data flow diagram illustrating an example surface feature detection pipeline according to some embodiments of the present disclosure;

[0015] Figure 4 is a data flow diagram illustrating an example truth data generation pipeline according to some embodiments of the present disclosure;

[0016] Figure 5 is a diagram illustrating an example ground model according to some embodiments of the present disclosure;

[0017] Figure 6 An example technique for sampling from a ground model according to some embodiments of the present disclosure is shown;

[0018] Figure 7 is an illustration of example boundaries of a navigable space according to some embodiments of the present disclosure;

[0019] Figure 8 is an illustration of an example navigable space and example extracted road hazards according to some embodiments of the present disclosure;

[0020] Figure 9 is a flow chart of a method for hazard detection according to some embodiments of the present disclosure;

[0021] Figure 10 is a flowchart of a method for generating a ground truth representation of one or more static hazards according to some embodiments of the present disclosure;

[0022] Figure 11 is a flow chart of a method for surface feature detection according to some embodiments of the present disclosure;

[0023] Figure 12 is a flow chart illustrating a method for generating a ground truth representation of one or more features of a detected ground surface, according to some embodiments of the present disclosure;

[0024] Figure 13A is an illustration of an example autonomous vehicle according to some embodiments of the present disclosure;

[0025] Figure 13B According to some embodiments of the present disclosure Figure 13A Examples of camera positions and fields of view for autonomous vehicles;

[0026] Figure 13C According to some embodiments of the present disclosure Figure 13A a block diagram of an example system architecture for an example autonomous vehicle;

[0027] Figure 13D is a method for one or more cloud-based servers and Figure 13A System diagram of an example of communication between autonomous vehicles;

[0028] Figure 14 is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure; and

[0029] Figure 15 is a block diagram of an example data center suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0030] Disclosed are systems and methods related to hazard detection and surface sensing in autonomous and semi-autonomous systems and applications. The present technology can be used to detect and navigate surfaces (e.g., roads, drivable, navigable, etc.), detect and avoid hazards, and / or generate corresponding ground truth data for various detection networks or other machine learning models (such as those used by autonomous vehicles, semi-autonomous vehicles, robots, and / or other types of objects or machines).

[0031] While the present disclosure may relate to an example autonomous or semi-autonomous vehicle or machine 1300 (alternatively referred to herein as "vehicle 1300" or "ego-machine 1300"), its examples relate to 13A to 13D For example, the systems and methods described herein may be used by, but are not limited to, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more advanced driver assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, aircraft, ships, space shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, airplanes, construction vehicles, trains, underwater vehicles, remotely controlled vehicles (such as drones), and / or other vehicle types. Furthermore, while the present disclosure may be described with respect to hazard detection or surface sensing for autonomous or semi-autonomous systems and applications, this is not intended to be limiting, and the systems and methods described herein may be used in augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications, and / or any other technology field in which object or surface detection may be used.

[0032] In some embodiments, the ego-machine may be equipped with one or more optical sensors (e.g., cameras) and one or more LiDAR sensors that may be used to generate corresponding image data and LiDAR data (e.g., as the ego-machine navigates an environment), and one or more neural networks may use transformers to fuse the image data and LiDAR data and detect one or more features of hazards and / or surfaces (e.g., roads or other navigable) in the environment.

[0033] Taking hazard detection as an example, an initial query including a set of reference or anchor 3D positions of a converter can be identified using candidate bounding shapes predicted from extracted image features and / or extracted LiDAR features, randomly initialized 3D positions, and / or ego-motion compensated converter predictions from the previous frame (e.g., for objects predicted with a threshold confidence). Extracted image features (e.g., in stereo) and extracted LiDAR features (e.g., in top-down or bird's-eye view (BEV)) can be sampled around each 3D reference point at keypoint locations identified by projecting learned (e.g., deformable) and / or specified 2D or 3D offsets into corresponding feature maps, and the sampled image and LiDAR features can be fused in a cross-attention layer of the converter. Thus, the converter can use the sampled image and LiDAR features to extract and decode a representation of whether a hazard is present at the 3D position corresponding to each of the initial query, its shape, and / or its class. For example, for the 3D reference point represented by each initial query, the output layer of the transformer may regress a representation of a two-dimensional (2D) or 3D bounding box (or other bounding shape) predicted to contain the detected object, may regress a representation of the uncertainty in the regressed bounding shape, and / or may classify the detected object into any number of supported classes (e.g., generating corresponding class confidence scores, such as a binary classification score indicating whether a predicted road fragment is present at the corresponding 3D location).

[0034] Additionally or alternatively, the converter can be used to generate a representation of one or more features of a specified portion of a surface (e.g., a road or other navigable surface). For example, a desired surface (e.g., a road) can be modeled using a set of sampled 3D points, and the converter can be used to predict one or more features of each point (e.g., precise height, distance, driving condition, etc.). In some embodiments, a 2D trajectory of the ego machine (e.g., one or more tire tracks) can be predicted (e.g., based on wheel angles), a set of 2D points can be sampled along each trajectory (e.g., logarithmically), and an initial height (e.g., zero in the equipment coordinate system) can be assigned to generate corresponding 3D reference points. Thus, an initial query for the converter can be identified using a set of sampled 3D reference points that model the specified portion of the surface and / or using converter predictions from the previous frame using ego motion compensation (e.g., for sampled points predicted with a threshold confidence level). The converter can use the sampled image and LiDAR features to extract and decode a representation of one or more features of the surface at the 3D location corresponding to each initial query. For example, for the 3D reference point represented by each initial query, the output layer of the transformer can regress a representation of the refined height value, can regress a representation of the uncertainty in the regressed height value, can regress a representation of the driving condition at the point (e.g., quantifying damage to the surface caused by the detected surface or weather conditions), and / or can classify the point into any number of supported classes (e.g., generating corresponding class confidence scores).

[0035] In some embodiments of detecting hazards and surface features, separate neural networks or the same multi-task network with corresponding transformer input and output heads can be used to detect hazards and surface features. In some embodiments implementing a multi-task network, the uncertainty of the predictions for the two tasks can be used as a coefficient to sum their losses during training.

[0036] In some embodiments, ground truth data for a neural network used to detect hazards and / or surface features or perform some other task can be generated in various ways. For example, one or more data collection vehicles can be equipped with one or more LiDAR sensors (e.g., a single roof-mounted 360° field-of-view LiDAR scanner), and the LiDAR sensors of the data collection vehicles (and / or fixed LiDAR sensors) can be used to collect LiDAR data frames representing various hazards and / or surface conditions. The LiDAR data can be self-motion compensated, any known techniques can be used to detect and regress the shapes of any specified class of dynamic obstacles represented in the LiDAR data, and the representations of the detected dynamic obstacles (e.g., 2D or 3D bounding boxes or other bounding shapes) can be used as ground truth data for the dynamic obstacle detection task (e.g., in a hazard detection network that includes one or more auxiliary heads for predicting the position of an initial transformer query).

[0037] In some embodiments, detected LiDAR points belonging to dynamic obstacles can be removed, and (e.g., obtained) point clouds from multiple frames can be registered with each other to increase point density and generate a refined ego-motion estimate (which supports increased accuracy of downstream surface feature estimates). Any known technique can be used to estimate a representation of the ground based on LiDAR data (e.g., classifying points predicted to be on the ground and projecting them onto a grid and smoothing the projected values, etc.) and / or based on image data (e.g., using 3D reconstruction to generate an estimated representation of the ground), and the representation of the ground can be used as ground truth data for the ground detection task. Taking an example embodiment involving a surface feature detection network as an example, the surface feature detection network predicts features of a specified portion of the surface (e.g., a set of points sampled along one or more predicted trajectories), the specified portion of the surface can be sampled and used as the ground truth. In some embodiments, any known smoothing technique can be applied to the specified portion of the surface (or some area containing the specified portion, such as an area extending outward from the ego machine along the surface) to refine the accuracy of the ground truth data.

[0038] In some embodiments, detected LiDAR points can be labeled as static or non-static based on the consistency of multiple observations. For example, a projected image (e.g., a range image) can be generated using each frame (e.g., a spin) of LiDAR data, a detected LiDAR point can be projected into the projected image for multiple frames, and a consistency measure of presence and / or distance can be used to classify the point as static or non-static. Typically, static objects should appear in a certain threshold number or percentage of frames or spins (e.g., at least 50% of spins, accounting for occlusion), and the detected distance to static objects should not change by more than a certain threshold amount even when the sensor moves (e.g., the detected distance should not double from one frame to another). In some embodiments, presence and / or distance consistency can be quantified and used to derive a consistency measure for each LiDAR point, with points with a consistency measure below a specified threshold being classified as non-static (e.g., a non-static part of a scene), while points with a consistency measure above a specified threshold being classified as static (e.g., part of a static object or a static part of a scene).

[0039] In some embodiments, representations of points and ground that are classified as static can be used to generate a representation of a navigable space (e.g., free space). A static point may represent the ground (in which case the height of the point represents the ground height) or a static hazard (in which case the height above the ground can be derived from the height of the point and the estimated ground). In some embodiments, the representation of the ground encodes a ground height (or corresponding distance) value. Thus, the ground height can be subtracted from the height of each static point to derive an estimated ground height for each static point. In addition, the ground can be represented as a grid with corresponding ground height or distance values, and the ground height or distance values ​​can be used to derive a corresponding surface curvature and assign a corresponding surface curvature to each grid cell. Thus, static points can be projected onto a grid, aggregated by cell, and compensated for noise, and an occupancy grid can be generated by evaluating the resulting ground height and corresponding estimated surface curvature for each cell using any known occupancy scoring function based on height and curvature. The resulting occupancy scores can be segmented into binary values ​​using a global cost minimization technique (e.g., classifying each grid cell as potentially occupied space or free space), and any known technique can be used to extract closed 2D contours represented in the resulting binary segmentation map.

[0040] Thus, a parent contour that encloses one or more sample points associated with the trajectory of the ego machine can be identified as a boundary of the navigable space (e.g., a free space boundary) and can be used to derive ground truth data for any navigable or free space segmentation task. For example, a contour can be assigned a corresponding estimated ground height and projected into a corresponding view to generate the boundary of a ground truth segmentation mask. The extracted child contours of the parent contour can be assumed to represent static hazards, assigned corresponding heights, projected into a ground truth segmentation mask, and used to carve out regions from the ground truth navigable space. In some embodiments of a hazard detection network involving initial converter queries using 3D reference points, (e.g., random) initial converter queries that are outside the ground truth navigable space can be omitted during training (e.g., to avoid training the network on regions predicted to be occupied by static hazards (such as potholes or other surface deformities) where automatically generated labels may be unreliable).

[0041] In some embodiments, the extracted sub-contours of the parent contour representing the predicted true navigable space (e.g., contours within the detected navigable space that do not belong to the navigable space) can be used to identify true static obstacles. Depending on the use case and / or embodiment, the extracted sub-contours that are longer than a specified length and / or represent or enclose LiDAR points below the ground can be filtered out. It can be assumed that the (e.g., remaining) sub-contours represent static obstacles, and 2D and / or 3D bounding boxes or other bounding shapes can be generated (e.g., using the maximum and minimum heights of the LiDAR points enclosed by each contour). Thus, the resulting representation of static obstacles can be used as ground truth data for any static obstacle detection task (e.g., in a hazard detection network).

[0042] In some embodiments, points belonging to detected dynamic obstacles are filtered out and remaining non-static points in the scene are identified, which can belong to weather or road conditions, such as rain or snow in the air or on the ground. Thus, in some embodiments, to support a weather condition detection task, a set of non-static scene points can be sampled from an area that is unlikely to contain any obstacles, such as a cube (e.g., one meter in length) or other volume in front of the ego machine and above the ground. Non-static scene points detected within this area (e.g., within a certain specified duration or number of frames) can be accumulated, and the resulting power distribution of the non-static scene points over different frequency components can be quantified (e.g., using power spectral analysis). Thus, power levels (or changes in power levels) at specified frequencies can be assigned labels representing corresponding types of weather conditions (e.g., rain, snow, fog, dust, clear). Additionally or alternatively, corresponding levels of impairment of visibility can be classified (e.g., using a linear relationship between the number and / or power of non-static scene points and corresponding classes of impairment of visibility). To support a driving condition detection task, similar procedures can be used to identify and / or accumulate non-static scene points from an area on the ground (e.g., in front of the ego machine), assign labels representing corresponding types of surface conditions (e.g., wet, snowy, icy, wet, dry), and / or assign labels representing corresponding levels of impairment of the surface. Thus, resulting weather and / or surface condition labels can be used as ground truth for any weather or road or road surface condition detection network.

[0043] In general, artifacts such as particles in the ambient weather (e.g., snowflakes) or transient events (e.g., condensations from a spout plume) can negatively impact the performance of time-of-flight sensors such as LiDAR sensors, e.g., by reducing the visibility of actual obstacles that the ego machine should avoid, or by appearing like real obstacles that need to be avoided. Thus, LiDAR points that are classified as part of the non-static scene rather than part of a detected dynamic obstacle can be flagged as artifacts. In some embodiments, points identified as part of a dynamic object can be subtracted from points identified as non-static, and the remaining points can be flagged as artifacts. Thus, artifact labels can be used as ground truth for any artifact detection task.

[0044] Thus, the techniques described herein can be used to detect hazards such as road debris and other obstacles, detect surface features such as ground height along a predicted trajectory, and / or generate ground truth data for various autonomous vehicle and / or other sensing tasks. Representations of the detected hazards and / or surface features can be provided to an autonomous vehicle driving stack to enable safe and comfortable planning and control of the autonomous vehicle. For example, the autonomous vehicle can navigate the vehicle to avoid detected hazards on the road or detected bumps in the road (e.g., potholes), adjust the vehicle's suspension system to match a detected road curve (e.g., by compensating for bumps in the road), and / or apply advance acceleration or deceleration based on an approaching surface slope in a detected road curve. Any of these functions should help enhance safety, increase vehicle life, improve energy efficiency, and / or provide a smooth driving experience.

[0045] refer to Figure 1 , Figure 1 is an example object detection pipeline 100 according to some embodiments of the present disclosure. It should be understood that this arrangement and other arrangements described herein are presented as examples only. In addition to the arrangements and elements shown, other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used, and some elements may be omitted entirely. In addition, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in combination with other components, and can be implemented in any suitable combination and location. The various functions performed by the entities described herein can be performed by hardware, firmware, and / or software. For example, the various functions can be implemented by a processor executing instructions stored in a memory. In some embodiments, the systems, methods, and processes described herein can be used in conjunction with 13A to 13D Example of autonomous vehicle 1300, Figure 14 The example computing device 1400 and / or Figure 15 The example data center 1500 may be performed using components, features, and / or functions similar to the components, features, and / or functions of the example data center 1500.

[0046] exist Figure 1 In the illustrated embodiment, the object detection pipeline 100 uses image data 101 and LiDAR data 121 (e.g., using a 13A to 13D1300 in the autonomous vehicle 1300) to detect one or more objects of a specified class in the environment. More specifically, the query generator 140 can use the candidate bounding shapes predicted by the auxiliary head 115 from the extracted image features 110 and / or the candidate bounding shapes predicted by the auxiliary head 135 from the extracted LiDAR features 130, use randomly initialized 3D positions, and / or use the (e.g., top K) ego-motion compensated converter predictions from the previous frame (e.g., for objects predicted with a threshold confidence) to generate a set of object queries 145, the set of object queries 145 including a set of reference or anchor 3D positions. A feature sampler 150 may sample image features 110 (e.g., in a stereo view) and / or LiDAR features 130 (e.g., in a top-down or bird's-eye view) around each 3D reference point at keypoint locations identified by projecting learned (e.g., deformable) and / or specified 2D or 3D offsets into corresponding feature maps, and a transformer (e.g., comprising an input layer 165, a transformer decoder 170, and one or more output heads 175) may use the sampled image and LiDAR features to extract a representation of whether an object (e.g., a hazard) is present at a 3D location corresponding to each of the object queries 145, its shape, and / or its class.

[0047] In some embodiments, a self-machine (e.g., 13A to 13D The autonomous vehicle 1300 in FIG. 1 may be equipped with one or more optical sensors (e.g., cameras, such as Figure 13A stereo camera 1368, wide angle camera 1370, infrared camera 1372, surround camera 1374, and / or long-range and / or mid-range camera 1398) and one or more LiDAR sensors (e.g., Figure 13A In the example of the LiDAR sensor 1364 in the ego machine, the optical sensor and the LiDAR sensor can be used to generate image data 101 and LiDAR data 121, respectively (e.g., when the ego machine navigates the environment), and the image data 101 and the LiDAR data 121 can be applied to corresponding input branches of the object detection pipeline 100. Sensor data from any given sensor can be generated at any frame rate, synchronized or otherwise associated with sensor data from other sensors, and processed by the object detection pipeline 100 at any frame rate. Sensor data from different sensors can be synchronized, but need not be synchronized. For example, the sensor data can be generated and / or aligned with respect to a common clock such that asynchronous sensor data from different sensors can be grouped together to represent a substantially common time slice. Thus, the techniques described herein can be used with synchronous and / or asynchronous sensor data. Figure 1The implementation shown in is by way of example only, and other embodiments may additionally or alternatively include input branches for other types of sensor data, such as RADAR data, sonar data, depth data, and / or other types.

[0048] The image data 101 may include frames from each of any number of optical sensors (e.g., cameras) and may be applied to an image encoder 105 (e.g., corresponding channels of the image encoder 105) to extract image features 110 using any known technique. For example, the image encoder 105 may include any number of channels including one or more corresponding (e.g., neural network, convolutional neural network (CNN)) layers that extract (e.g., multi-scale) image features from the corresponding optical sensors (e.g., cameras). Hyperparameters such as kernel size, stride, channel count, and / or number of repetitions may be selected to balance accuracy and speed. The image encoder 105 may use upsampling to merge coarser feature maps with finer feature maps.

[0049] In some embodiments, the image features 110 are processed by the auxiliary head 115 to generate corresponding predictions (e.g., 2D or 3D object detections, dense depth), and these predictions can be used to generate object queries 145 and / or generate auxiliary losses to help train the image encoder 105. For example, the object queries 145 can include a set of initial 3D queries that represent locations and / or regions where the auxiliary head 115 predicts that objects may be present, and the converter can effectively refine these predictions. Depending on the embodiment, the auxiliary head 115 can generate different types of predictions, which the query generator 140 can use to generate initial 3D queries for the converter. For example, the auxiliary head 115 can generate (and the query generator 140 can decode) predictions of one or more specified classes (which can correspond to the classes predicted by the converter) and / or (e.g., dense) depth estimates, such as 2D or 3D bounding boxes or other bounding shapes. Thus, the query generator 140 can use the predicted bounding shapes and / or corresponding depth estimates to generate one or more initial 3D queries.

[0050] For example, the query generator 140 may sample any number of 2D points (e.g., one representative 2D point, such as the center point of each proposed 2D bounding shape) from one or more predicted 2D bounding shapes (e.g., the top M with the highest prediction confidences), and may sample any number of corresponding 3D points for a projected 3D representation of each sampled 2D point. For example, the query generator 140 may back-project the sampled 2D points into 3D using corresponding (extrinsic and intrinsic) calibration parameters of the corresponding sensor, and sample (e.g., ten points) in 3D along a 3D ray pointing from an origin (e.g., the origin of the device coordinate system) to the back-projected 3D position of the sampled 2D point. Additionally or alternatively, the query generator 140 may use the predicted depth to estimate how far along 3D space the detected object is, and may sample the 3D ray at one or more depths selected from the corresponding depth map at one or more pixels corresponding to one or more 2D points (e.g., the center point, one or more corners) from the predicted 2D bounding shape. In some embodiments, the auxiliary head 115 generates a 3D bounding shape, and the query generator 140 can sample one or more 3D points (e.g., a center 3D point) from one or more predicted 3D bounding shapes (e.g., the top M with the highest prediction confidence). These are just a few examples, and other ways of sampling from predicted 2D and / or 3D bounding shapes can be implemented within the scope of the present disclosure.

[0051] Returning to the LiDAR data 121, the LiDAR data 121 can include LiDAR detections from any number of LiDAR sensors and any number of spins or scans. In some embodiments, LiDAR processing 125 can be used to aggregate the LiDAR data 121 from several spins or scans to create a more comprehensive and detailed LiDAR point cloud, estimate a refined ego-motion representing the trajectory of the ego machine, and use the refined ego-motion to ego-compensate the LiDAR data 121 and align it to a common reference frame. Thus, the LiDAR data (e.g., the LiDAR data 121, the aggregated and ego-motion compensated LiDAR data) can be applied to the LiDAR encoder 127 to extract LiDAR features 130 using any known technique (such as point cloud segmentation, projecting the 3D point cloud into a 2D view and then evaluating the resulting 2D projected image) or using a mapping algorithm (such as simultaneous localization and mapping (SLAM)). In an example embodiment, the LiDAR encoder 127 may discretize or bin the LiDAR detections into columns or columns corresponding to cells of a 2D grid, encode the points in each column or column (e.g., using PointNet or a related architecture), fill the encoded features into corresponding cells of the 2D grid to generate a pseudo image (e.g., in a bird's-eye view), and extract LiDAR features 130 from the pseudo image using one or more (e.g., a neural network, such as a CNN) layers.

[0052] In some embodiments, the LiDAR features 130 can be processed by the auxiliary head 135 to generate corresponding predictions (e.g., 2D or 3D object detections), and these predictions can be used to generate object queries 145 and / or generate auxiliary losses to help train the LiDAR encoder 127. For example, the auxiliary head 135 can generate (and the query generator 140 can decode) predictions of one or more specified classes (which can correspond to the classes predicted by the converter), such as 2D or 3D bounding boxes or other bounding shapes, and the query generator 140 can use the predicted bounding shapes to generate one or more initial 3D queries. As a non-limiting example, the auxiliary head 135 can generate 3D bounding shapes (or 2D bounding shapes and height above ground, which can be used to generate 3D bounding shapes), and the query generator 140 can sample one or more 3D points (e.g., center 3D points) from one or more predicted 3D bounding shapes (e.g., the top M with the highest prediction confidence).

[0053] In some embodiments, a transformer (e.g., comprising an input layer 165, a transformer decoder 170, and one or more output heads 175) can be used to generate a representation of whether an object (e.g., a hazard) is present at a 3D location corresponding to each object query 145, its shape, and / or its class. Depending on the embodiment, the query generator 140 can generate any number and type of transformer queries that represent a set of reference or anchor 3D locations (e.g., candidate object locations). For example, the query generator 140 can identify the 3D location of the object query 145 using candidate bounding shapes predicted by the auxiliary head 115 based on the image features 110 and / or candidate bounding shapes predicted by the auxiliary head 135 based on the LiDAR features 130, using randomly initialized 3D locations, and / or using (e.g., the top K) ego-motion compensated transformer predictions from the previous frame (e.g., for objects predicted with a threshold confidence level). As a limiting example, the input layer 165 of the transformer may accept a representation of 350 queries, which the query generator 140 may populate using 50 3D queries predicted by the transformer from the previous frame, 50 3D queries of bounding shape samples predicted from image features 110, 50 3D queries of bounding shape samples predicted from LiDAR features 130, and 200 randomly sampled 3D queries (e.g., selected from a uniform distribution). Thus, object queries 145 may be applied to the input layer 165 of the transformer, which may transform the object queries 145 into corresponding embeddings (e.g., by projecting the 3D coordinates into a higher dimensional space that matches the input dimensionality expected by the transformer decoder 170, adding a position encoding to the embedding to provide spatial or temporal context). Thus, the object queries 145 may be used as candidate locations to be evaluated by the transformer.

[0054] In some embodiments, rather than considering all possible combinations of query and image and LiDAR feature elements (which can be computationally expensive and inefficient for high-resolution input), the attention mechanism of transformer-decoder 170 can focus on a relevant subset of locations sampled around a reference 3D location represented by each object query 145. This sampling can be conceptualized as a set of offsets around a reference point, effectively deforming a grid of attention locations based on the (e.g., learned or fixed) offsets. Consequently, transformer-decoder 170 can compute attention weights for each of these sampled embeddings, where these weights determine how much influence each sampled point has on updating the representation of the object query 145. Focusing on a sparse set of sampled points can reduce computational complexity compared to traditional dense attention mechanisms.

[0055] More specifically, the feature sampler 150 may sample image features 110 (e.g., in stereo view) and / or LiDAR features 130 (e.g., in top-down or bird's-eye view) around a 3D reference point represented by each object query 145 at keypoint locations identified by projecting learned (e.g., deformable) and / or specified 2D or 3D offsets into corresponding feature maps, and may apply the sampled features to the converter decoder 170. Taking a query representing a reference 3D position as an example, the feature sampler 150 may project the reference 3D position into a 2D (e.g., stereo view) view represented by the image features 110 using corresponding intrinsic and extrinsic parameters for the corresponding optical sensor, and may project the reference 3D position into a 2D (e.g., top-down) view represented by the LiDAR features 130 using corresponding intrinsic and extrinsic parameters for the corresponding LiDAR sensor (e.g., updated to reflect refined ego-motion). In some embodiments, the feature sampler 150 may sample features from the feature map extracted from the sensor data of each sensor at 2D keypoint locations identified by applying one or more 2D offsets to the projected 2D locations in the corresponding extracted features. The 2D offsets may be learned or fixed and may vary for each query, feature map extracted (e.g., corresponding to different sensors), and / or attention heads to which the sampled features are applied. Additionally or alternatively, each 3D reference point represented by each object query 145 may be associated with a corresponding 3D keypoint location identified by applying one or more 3D offsets to the reference 3D location (e.g., the center of each of the six faces of the corresponding 3D bounding box plus the center of the 3D bounding box), and the feature sampler 150 may project the 3D keypoint into the extracted feature map and sample the extracted feature map at the projected location. The 3D offsets may be learned or fixed and may vary for each query, feature map extracted (e.g., corresponding to different sensors), and / or attention heads to which the sampled features are applied. Thus, the feature sampler 150 may sample any number of features from each extracted feature map (e.g., each channel of the image features 110, the LiDAR features 130) for each object query 145 and / or each attention head of the transformer decoder 170.

[0056] Thus, the transformer decoder 170 can use the encoded representation of the object query 145 and the sampled image and LiDAR features to detect objects (e.g., predict the coordinates of a bounding box or other bounding shape 180 for each object). The transformer decoder 170 can include any number of transformer blocks, where each transformer block can include a self-attention layer and a cross-attention layer. Each self-attention layer can include any number of attention heads that compute attention scores representing the relationship between each query and each other query and convert these scores into attention weights (e.g., using a softmax function) that determine how much attention each query should pay to each other query. The attention weights can be used to create a weighted sum of values ​​associated with the queries, thereby refining the representation of the object query 145. In embodiments with multiple attention heads, each head can perform these operations independently, and the results can be concatenated and linearly transformed to generate a refined representation of the object query 145. This self-attention mechanism allows queries to share information with each other, thereby helping the transformer understand the global context and dependencies between potential objects.

[0057] Each crisscross attention layer can include any number of attention heads that compute attention scores representing the relationship between each query and each sampled feature embedding and convert these scores into attention weights (e.g., using a softmax function) that determine how much attention each query should pay to each sampled feature embedding. The attention weights can be used to create a weighted sum of the sampled feature embeddings, thereby effectively fusing sensor data from different sensors and further refining the representation of the object query 145 to represent the combination of each object query 145 and the sampled features. In an embodiment with multiple attention heads, each head can perform these operations independently, and the results can be concatenated and linearly transformed to generate a combined representation of each object query 145 and the sampled features. This crisscross attention mechanism allows the query to integrate information from the sampled features, helping the converter understand the global context and the dependencies between potential objects.

[0058] Thus, the transformer decoder 170 may include any number of transformer blocks that iteratively refine the combined representation of each object query 145 and the sampled features. For example, the transformer decoder 170 may output a vector for each object query that may be applied to one or more output heads 175 (e.g., an output head for each of a plurality of specified classes) that regress a representation of a 2D or 3D bounding box or other bounding shape predicted to contain the detected object anchored at the 3D reference point represented by the object query, regress a representation of the uncertainty in the regressed bounding shape, and classify the detected object into any number of supported classes (e.g., generating corresponding class confidence scores, such as a binary classification score indicating whether a predicted road debris is present at the corresponding 3D location).

[0059] In some embodiments, output head 175 includes N channels (e.g., classifiers), where each channel regresses a representation of a particular aspect of the size, shape, or position of a detected object (e.g., from a particular class, for all classes, etc.), such as where the object is located relative to a 3D reference point (e.g., a dx / dy vector pointing to a part of the object (such as the center or a corner)), object height, object width, object orientation (e.g., a rotation angle such as sine and / or cosine), some statistical measures thereof (e.g., minimum, maximum, mean, median, variance, etc.), uncertainty in one or more aspects of the regression information, etc. Thus, output head(s) 175 can be used to predict regression data representing a 2D or 3D bounding box or other bounding shape anchored at a 3D reference point represented by the corresponding object query.

[0060] Additionally or alternatively, the output head 175 may include a channel (e.g., a classifier) ​​for each class of objects to be detected (e.g., vehicles, cars, trucks, vulnerable road users, pedestrians, cyclists, motorcycles, static hazards such as road debris, some subclass thereof, etc.), where each channel performs one or more classifications (e.g., a classification score quantifying the likelihood that a specified object class is located at a 3D reference point represented by a corresponding object query, a binary classification score, etc.).

[0061] In some embodiments, the alignment component 190 may identify a specified number of predictions (e.g., the top K with the highest prediction confidences) and add them to the object query 145 for the next time step. For example, one or more output heads 175 of the transformer may generate a vector, tensor, or other data structure representing prediction parameters (e.g., a classification score for each object query (e.g., for each of one or more classification channels)), and the alignment component 190 may read the classification scores for the object query 145, identify a specified number of the highest scores, read and / or decode the prediction parameters representing the predicted positions of the corresponding objects, update the predicted positions using the refined ego-motion to reflect the ego-motion of the current frame and align the predictions with the current position and orientation of the ego-machine, and provide the updated positions to the query generator 140 for inclusion in the object query 145 for the next prediction.

[0062] Thus, the converter formed by one or more input layers 165, the converter decoder 170, and the one or more output heads 175 can iteratively refine the object prediction. Each layer of the converter can adjust the predicted position and features of the predicted object based on the fused image and LiDAR features, thereby improving the detection accuracy at each iteration. Thus, the converter can output a representation of the predicted position, size, and / or class of the object detected in the 3D environment. The object detection can be used by one or more control components of the autonomous vehicle, such as the controller 1336, the ADAS system 1338, and / or an autonomous driving software stack (such as the stack described in U.S. Patent Application No. 16 / 938,706, Publication No. 20210026355A1) executed on one or more components of the vehicle 1300 (e.g., SoC 1304, CPU 1318, GPU 1320, etc.). For example, the parameters predicted by the converter can be decoded to generate a 3D bounding box or other bounding shape 180 and corresponding class labels and confidences, these object detections can be provided to the control component, and the control component can use any known techniques to use the object detections to navigate, plan, or otherwise perform one or more operations within the environment (e.g., obstacle avoidance, lane keeping, lane changing, merging, lane splitting, etc.).

[0063] Furthermore, it is often useful (e.g., for autonomous vehicles) to understand the elevation profile of a road or other surface. The elevation profile of a road or other navigable surface can provide information about elevation changes, slopes, and potential inclines or descents along a route, which can be used to optimize speed, braking, and acceleration. Furthermore, the vertical profile can be used to improve passenger comfort by anticipating and smoothly handling hills or uneven terrain. Figure 2An example elevation curve 230 of a road surface according to some embodiments of the present disclosure is shown. For example, as vehicle 220 travels on road 210, the elevation of road 210 ahead of vehicle 220 may vary due to topographical features such as hills and valleys, expansion joints of bridges, steps or milled edges at construction sites, potholes, lane grooves or ruts, speed bumps, cracks in the road surface, rough gravel resulting from the stripping of asphalt or smooth paving, joints between concrete slabs, or other curvature or damage to the road or other driving surface. Therefore, it may be useful to detect the elevation of the road or other features (e.g., uncertainty in predicted elevation 240) at one or more points ahead of the vehicle, such as points along the trajectory of vehicle 220, such as the trajectory of one or more tires (e.g., trajectory 250).

[0064] Figure 3 is a data flow diagram illustrating an example surface feature detection pipeline 300 according to some embodiments of the present disclosure. The components of the surface feature detection pipeline 300 may have corresponding functions, which components use Figure 1 100 (e.g., image encoder 105, LiDAR encoder 127, feature sampler 150). Thus, a related architecture can be used to detect one or more features of a road or other surface in an environment. In the surface feature detection pipeline 300, a query generator 340 can generate a set of surface queries 345 using one or more ego-machine tracks (e.g., one or more tire tracks) predicted by a path generator 350 based on corresponding wheel angles 337 and / or ego-motion compensated transformer predictions from a previous frame (e.g., for surface locations predicted with a threshold confidence level). Thus, a transformer formed of one or more input layers 365, a transformer decoder 370, and one or more output heads 375 can use sampled image and LiDAR features to extract representations of one or more surface features 380 (e.g., height, distance, driving condition, etc.) at 3D locations corresponding to each surface query, and can iteratively refine the surface queries 345 representing candidate 3D locations on the surface to detect one or more curves of the surface.

[0065] Depending on the downstream use case, different surface features and different portions of the surface may be of interest. For example, some embodiments may attempt to detect representations of height curves of a road or other surface on the tire track ahead of the vehicle. Thus, the query generator 340 may sample an arbitrary number of 2D or 3D points along the predicted track (e.g., sampling 2D points along one or more tire tracks in a bird's-eye view and assigning a candidate height (such as zero) to each point). This is merely an example, and other techniques may be used to sample candidate points of the road or other surface (e.g., sampling a specified number of points from a specified plane (such as z=0) or some other specified area in the environment).

[0066] Continuing with the example of sampling one or more 2D or 3D points along one or more predicted trajectories, path generator 350 can identify trajectories based on wheel angles 337. For example, path generator 350 can use an Ackermann steering model and wheel angles 337 to generate a representation of the trajectories of one or more tires by simulating the vehicle's kinematic behavior based on the vehicle's steering geometry. The Ackermann model defines a geometry that indicates how a vehicle's wheels should tilt during a turn so that the vehicle follows a smooth trajectory. Using this model, the vehicle's inner and outer wheels should follow circular paths with different radii centered around a common turning point during a turn. Thus, steering angle sensors and / or wheel position sensors can be used to detect wheel angles 337 of the left and right tires, and path generator 350 can use the wheel angles 337 of the left and right tires and the Ackermann model to calculate a representation of the predicted trajectory for each tire, such as the turning radius and curvature of each path in a 2D (e.g., top-down) view of the vehicle's plane of motion (e.g., the xy plane), assuming a flat road surface. The query generator 340 can use the representation of each predicted trajectory to sample an arbitrary number of 2D points (e.g., at regular intervals, logarithms, etc.) along each trajectory (e.g., along an arc length extending from the front wheel to a specified distance (such as 15 meters)) and assign an initial height (e.g., zero) to each 2D point to generate a corresponding candidate 3D position. In this way, the query generator 340 can identify surface queries 345 using a set of sampled 3D reference points that model a specified portion of the surface and / or using ego-motion compensated converter predictions from a previous frame (e.g., for sampled points predicted with a threshold confidence level).

[0067] The converter may use the sampled image and LiDAR features to extract and decode representations of surface features 380 at the 3D location corresponding to each surface query 345 . Figure 3 The input layer 365 of the converter and / or the converter decoder 370 may have Figure 1The input layer 165 of the transformer and / or the transformer decoder 170 may function similarly, but may adapt to the dimensionality of the surface query 345. The output head 375 may use the sampled image and LiDAR features to extract a representation of the surface features 380. For example, the transformer decoder 370 may output a vector for each surface query, and the vector for each surface query may be applied to one or more output heads 375 (e.g., an output head for each of a plurality of specified classes), which may regress a representation of the refined height of the surface at the 3D location corresponding to the surface query, regress a representation of the uncertainty in the regressed height, regress a representation of the driving condition at the 3D location (e.g., quantifying damage to the surface caused by the detected surface or weather conditions), and / or classify the point into any number of supported classes (e.g., generating a corresponding class confidence score).

[0068] In some embodiments, the output head 375 includes N channels (e.g., classifiers), where each channel regresses a representation of a particular feature of the surface at the 3D reference point requested by the surface query (e.g., height above the ground, distance, a quantified representation of damage to the surface caused by detected surface or weather conditions, slip resistance, surface gradient or curvature, etc.), uncertainty in one or more aspects of the regressed information, etc. Thus, the output head 375 can be used to predict regression data representing one or more features of the surface at the 3D reference point represented by the corresponding surface query.

[0069] Additionally or alternatively, the output head 375 can include a channel (e.g., a classifier) ​​for each class of surface features to be detected (e.g., surface condition, such as cracked, potholed, or smooth; presence of road markings; texture type, such as grooved or rough; contamination type, such as debris, water, or oil; and / or other), where each channel performs one or more classifications (e.g., a classification score quantifying the likelihood that a specified class of surface features is located at a 3D location represented by a corresponding surface query, a binary classification score, etc.).

[0070] In this way, the transformer formed by one or more input layers 365, transformer decoders 370, and one or more output heads 375 can iteratively refine the predicted surface features 380. Each layer of the transformer can adjust the predicted position and characteristics of the predicted surface features 380 based on the fused image and LiDAR features, thereby improving the detection accuracy with each iteration. The transformer can output a representation of the predicted surface features 380 in the 3D environment, which can be used by one or more control components of the autonomous vehicle, such as the controller 1336, the ADAS system 1338, and / or an autonomous driving software stack (such as the software stack described in U.S. Patent Application Publication No. 20210026355A1) executed on one or more components of the vehicle 1300 (e.g., SoC 1304, CPU 1318, GPU 1320, etc.). For example, the parameters predicted by the converter can be decoded to extract the height and / or uncertainty values ​​for each sample point on the surface, the height and / or uncertainty values ​​can be provided to one or more control components, and the one or more control components can use the height and / or uncertainty values ​​to navigate, plan, or otherwise perform one or more maneuvers within the environment (e.g., obstacle or bump avoidance, lane keeping, lane changing, merging, lane splitting, adjusting the suspension system of the ego object or ego actor to match the current road surface, applying advance acceleration or deceleration based on an approaching road surface slope, mapping, etc.) using any known technique.

[0071] In general, any suitable loss function can be used for training Figure 1 The object detection pipeline 100 and / or Figure 3 Surface feature detection pipeline 300 is used to compare one or more predicted outputs with the true value. Example loss functions for regression output include mean squared error (MSE), mean absolute error (MAE), Huber loss, and smoothed L1 loss, and example loss functions for classification output include cross entropy loss, local loss, and hinge loss, to name a few. In some embodiments, Figure 1 The object detection pipeline 100 and / or the surface feature detection pipeline 300 in

[0045] employ one or more auxiliary losses (e.g., based on Figure 1 The output of the auxiliary head 115 and / or 135, Figure 3 The output of the auxiliary loss 315 and / or 335 is trained to help train the image encoder 105 and / or the LiDAR encoder 127. In some scenarios, when training the object detection pipeline 100, (e.g., random) initial transformer queries that are outside the ground-truth navigable space can be omitted (e.g., to avoid training the object detection pipeline 100 on areas predicted to be occupied by static hazards (such as potholes or other surface deformities), where automatically generated labels may be unreliable). In some embodiments, Figure 1 The object detection pipeline 100 and Figure 3 The surface feature detection pipeline 300 can be combined into a multi-task network with a common transformer-decoder and corresponding input and output heads for object and surface feature detection. In some embodiments implementing a multi-task network, the uncertainty of the predictions for the two tasks can be used as a coefficient to sum their losses during training.

[0072] In some embodiments, a method for training a neural network (e.g., Figure 1 Object detection pipeline in 100, Figure 3 Ground truth data for parameter adjustment of a classical machine learning model for detecting objects and / or surface features) and / or for surface feature detection pipeline 300 in

[0065] and / or some other neural network and / or for detecting objects and / or surface features can be generated in a variety of ways. For example, some embodiments may employ automated methods (e.g., using classical non-machine learning algorithms) to derive various types of ground truth data from sensor data collected by one or more data collection vehicles. In example embodiments, a high degree of automation may be achieved using sensor data from a single LiDAR sensor (e.g., a 360-degree field of view, roof-mounted LiDAR scanner), but in general, any number of sensors may be used to collect the underlying sensor data.

[0073] Figure 4 4 is a data flow diagram illustrating an example ground truth data generation pipeline 400 according to some embodiments of the present disclosure. According to embodiments, ground truth data, such as data representing dynamic obstacles 420, (e.g., ground) surface models 435, surface features 445, static scene points 455, navigable space boundaries 465, hazardous objects 475, non-static scene points 480, weather and / or surface condition labels 485, and / or per-point artifact labels 495, can be derived using one or more paths through the ground truth data generation pipeline 400.

[0074] In some embodiments, one or more data collection vehicles may be equipped with one or more LiDAR sensors (e.g., a single roof-mounted 360° field-of-view LiDAR scanner; a forward-facing, grille-mounted, or windshield-mounted long-range LiDAR sensor; etc.), and the data collection vehicle's LiDAR sensors (and / or fixed LiDAR sensors) may be used to collect frames of LiDAR data 402 representing various objects (e.g., hazards) and / or surface conditions as the data collection vehicle navigates the environment. In some embodiments where image data is used to support detection tasks, the data collection vehicle may be equipped with one or more cameras for collecting corresponding frames of image data. Depending on the desired use case for the ground truth data, environments and / or scenarios may be selected or specified to cover a range of conditions, terrains, weather conditions, times of day, traffic densities, and / or road types to ensure comprehensive data collection. In some cases, target objects and / or surface features may be simulated (e.g., using plywood of increasing thickness to simulate a step edge in a road surface), and sensor data supporting different detection tasks (e.g., object and surface feature detection) may be collected during the same data collection capture run. The ego-motion 404 of the data collection vehicle may be recorded using any known technique (e.g., using an inertial measurement unit (IMU), a global positioning system (GPS), etc.). In some embodiments, LiDAR data collected using a data collection vehicle with one or more LiDAR sensors (e.g., multiple capture runs) may be combined with LiDAR data collected using a fixed 3D (e.g., LiDAR) scanner to improve coverage of target objects and / or surface features and increase point density in the area of ​​interest. While some embodiments operate on data collected by the data collection vehicle (e.g., after one or more data capture runs), this is not necessarily the case. For example, any ego machine (e.g., a production vehicle) may collect LiDAR data (e.g., using a forward-facing, grille-mounted, and / or windshield-mounted long-range LiDAR sensor) and execute part or all of the pipeline 400 in real time to generate a Figure 4 Any data depicted in , whether it is ultimately used as ground truth data, provided in some form to one or more downstream control components of the self-machine, and / or otherwise.

[0075] In some embodiments, motion compensation 410 can be applied to LiDAR data 402 from any number of LiDAR sensors, data collection runs, and / or spins or scans using the known ego-motion 404 of the data collection vehicle to transform the raw LiDAR distance measurements into a common spatial representation (e.g., an aggregated LiDAR data frame representing the scene in the environment). Additionally or alternatively, point cloud filtering 412 (e.g., on a per-spin basis) can be used to remove points with inconsistent distance measurements (e.g., exceeding a threshold difference) between the first and second returns (or, for example, between the first and strongest returns), points marked as invalid by the LiDAR sensor (e.g., based on the quality or reliability of the return signal), points with a distance greater than a specified threshold (e.g., 150 meters), and / or points marked as transient (e.g., rain, fog, snow, dust plumes) detected using any known technique (e.g., using one or more machine learning models). This pre-processing step can be used to improve the quality of the LiDAR point cloud by removing many artifacts that can adversely affect downstream operations (e.g., subsequent point cloud registration).

[0076] In some embodiments, dynamic obstacle detection 415 may use any known technique to detect and / or regress the 2D or 3D shape of any specified class of dynamic obstacles represented in the LiDAR data (e.g., point cloud segmentation, projecting a 3D point cloud into a 2D view, and then evaluating the resulting 2D projected image, as described, for example, in U.S. Patent No. 11,532,168). Example classes of dynamic obstacles include cars, trucks, buses, motorcycles, pedestrians, cyclists, animals, and the like. The LiDAR points enclosed by each detected 3D bounding box or other bounding shape (or the LiDAR points projected onto each detected 2D bounding box or other bounding shape) may be identified, and these LiDAR points may be tracked over a specified number of frames (e.g., three frames) to determine whether the identified points are moving. If so, the bounding shape may be identified and marked as a dynamic obstacle, and / or the corresponding point may be identified and marked as part of a dynamic obstacle. In some embodiments, dynamic obstacles 420 may be represented using a 3D bounding box or other bounding shape that describes the extent of each dynamic obstacle, and / or using some or all points contained in the bounding shape to represent dynamic obstacles 420. Example uses of dynamic obstacles 420 include online dynamic obstacle estimation (e.g., training Figure 1 The object detection pipeline 100 in the embodiment of the present invention is used to detect dynamic obstacles, and the auxiliary loss is calculated, such as the above object detection pipeline 100 or Figure 3 assisted losses as described in the surface feature detection pipeline 300 in FIG.

[0077] In some embodiments, detected LiDAR points belonging to dynamic obstacles may be removed, and point cloud registration 425 may be applied to register point clouds from multiple LiDAR spins or scans (e.g., filtered point clouds) with each other to increase point density and generate refined ego-motion estimates (which support increased accuracy of downstream surface feature estimates). Point cloud registration 425 may use any known technique, such as iterative closest point (ICP), normal distribution transform (NDT), or SLAM.

[0078] Estimating the ground surface is a prerequisite for many tasks in the field of autonomous driving, as the estimated ground surface can be used as a baseline for detecting obstacles on the road surface, can provide an estimate of the drivable area near the vehicle, and / or can be used as a guide for estimating the height of overhanging obstacles. Therefore, in some embodiments, ground surface estimation 430 can use any known technique to estimate a representation of a ground surface model 435 based on (e.g., obtained) LiDAR data (e.g., classifying points predicted to be on the ground surface and projecting them onto a grid and smoothing the projection values, for example, as described in U.S. Patent Application No. 17 / 992,569, Publication No. US20240028041A1) and / or based on image data (e.g., using 3D reconstruction to generate an estimated representation of the ground surface). The ground surface can be defined as a smooth and continuous surface that spans outward from the base or contact point of the vehicle, and ground surface estimation 430 can focus on estimating a single ground surface model and thus avoid the ambiguity involved in finding multiple non-intersecting surface models. The ground model 435 can be defined to cover the same range of distances as the LiDAR data (e.g., up to 300 meters) and can provide robust estimates for areas of the input data that are occluded by certain non-ground objects (e.g., people, vehicles, and other objects on the road). As a result of the ground estimate 430, each LiDAR point can be assigned a height above the ground. In some embodiments, the height above the ground can be used to extract a corresponding curvature estimate by analyzing the height variation between points (e.g., fitting a curve or surface to the local patch of points and calculating the rate of change of slope in one or more directions).

[0079] In some embodiments, ground estimation 430 may include iteratively refining the accuracy of ground model 435 by minimizing (or approximately minimizing) a cost function that penalizes deviations between measured and estimated values ​​(a data term) and / or deviations between adjacent measurements (a smoothing term), such as described in U.S. Patent Application No. 17 / 992,569. In some scenarios, there may be systematic measurement errors in the measured height values ​​of LiDAR points, depending on the distance and / or angle of the observation point. Therefore, to compensate for these measurement errors, weights may be applied to the measurements during the estimation process. For example, LiDAR points with shorter distances, observed at steeper angles, and / or with smaller beam coverage may be assumed to have higher accuracy and thus be assigned relatively higher weights, while LiDAR points with longer distances, observed at shallower angles, and / or with larger beam coverage may be assumed to have lower accuracy and thus be assigned relatively lower weights. By incorporating these weights into the cost function during iterative refinement of ground model 435, the influence of points with higher errors may be reduced, thereby achieving a more accurate and reliable ground estimate.

[0080] Ground model 435 can be represented using a scalar indicating the estimated height above ground for each point on the surface and / or a regularly spaced grid (e.g., spanning part or all of the scene), where the value of each grid cell encodes the height above ground level (AGL). Example uses of ground model 435 include ground truth for online ground estimation, ground truth for sensor self-calibration based on ground observations, and providing per-point classification as (a) ground point, (b) obstacle point (e.g., 0.1 to 2.5 meters AGL), and / or (c) head vertex (e.g., AGL>2.5 meters). Figure 5 is an illustration of an example ground model according to some embodiments of the present disclosure. More specifically, Figure 5 A stereogram of the fused point cloud is shown, with points above 2.5 meters above the ground shown in the lightest gray, points on the ground model shown in medium gray shades, and obstacle points up to 2.5 meters above the ground shown in the darkest gray.

[0081] In some embodiments, ground model 435 is used to generate a representation of ground truth surface features 445. Taking as an example an example embodiment of a surface feature detection network that involves predicting features of a specified portion of the ground (e.g., a set of points sampled along one or more predicted trajectories), sampling 440 can be applied to sample any number of points from the specified portion of ground model 435. Figure 6An example technique for sampling from a ground model according to some embodiments of the present disclosure is shown. For example, assuming that the portion of interest in ground model 610 is an area extending outward from a data collection vehicle along one or more tracks (e.g., track 620), one or more segments or regions (e.g., region 630) of ground model 610 that are proximate to the one or more tracks can be selected and sampled to obtain ground truth points. In some embodiments, selected points in the selected segments or regions of ground model 610 can be iteratively refined (e.g., as described above) and used to sample any number of points using any suitable sampling technique. Thus, feature encoding 442 can be used to encode features of the sampled points that are relevant to a downstream task or use case. Surface features 445 can be represented using: a scalar indicating a corresponding feature for each sampled point on the surface (e.g., height, distance, a quantitative representation of surface damage, such as those represented by surface condition label 485, surface gradient or curvature, etc.), and / or a regularly spaced grid (e.g., spanning part or all of the scene) where one or more values ​​for each grid cell encode the value of the corresponding feature. Example uses of surface features 445 include online surface feature estimation (e.g., by Figure 3 surface feature detection pipeline 300).

[0082] In some embodiments, point filtering 450 can be applied to label detected LiDAR points as static or non-static based on the consistency of multiple observations. For example, each frame (e.g., spin) of (e.g., filtered) LiDAR data can be used to generate a corresponding projection image (e.g., a range image) by projecting the LiDAR point into a corresponding 2D view. Any given LiDAR point can be projected into the projection images of multiple frames, and a consistency measure of presence and / or distance can be used to classify the point as static or non-static. For example, a consistency measure of presence and / or distance can be quantified based on whether a given LiDAR point appears in a certain threshold number or percentage of frames or spins (e.g., at least 50% of the spins, accounting for occlusions), and / or whether its detection range changes by more than a certain threshold amount when the sensor moves (e.g., the detected range should not double from frame to frame). In some embodiments, consistency measures of presence and / or distance can be quantified and used to derive a consistency measure for each LiDAR point. Points with consistency measures below a specified threshold can be classified as non-static (e.g., non-static parts of a scene), while points with consistency measures above the threshold can be classified as static (e.g., parts of static objects or static parts of a scene). This is by way of example only, and other space carving techniques can be applied in 2D or 3D to iteratively carve out areas of space that are unlikely to contain static objects (and corresponding static points). Static scene points 455 and / or non-static scene points 480 can be represented using a per-point binary indicator (e.g., a flag) assigned to each point in the dataset, the per-point binary indicator indicating whether the point is static or non-static. Example uses of static scene points 455 and / or non-static scene points 480 include generating a 3D voxel grid representing static occupancy (e.g., watertight) by assigning a binary value to each voxel, the binary value indicating whether the voxel is occupied (e.g., occupied by a solid object) or unoccupied (empty space).

[0083] In some embodiments, navigable space estimation 460 is applied to generate a representation of a navigable (e.g., free) space (e.g., navigable space boundary 465) using static scene points 455 and ground model 435. Static scene points 455 may represent the ground (in which case the point's height represents the ground height) or a static hazard (in which case the height above the ground can be derived from the point's height and ground model 435). In some embodiments, ground model 435 encodes ground height (or corresponding distance) values. Thus, the ground height can be subtracted from the height of each static scene point 455 to obtain an estimated height above the ground for each static scene point. Furthermore, ground model 435 can be represented as a grid with corresponding ground height and surface curvature values. Thus, static scene points 455 can be projected onto the grid, aggregated per cell, and noise compensated, and an occupancy grid can be generated by evaluating the resulting height above the ground and the corresponding estimated surface curvature for each cell using any known height- and curvature-based occupancy scoring function. The resulting occupancy scores can be segmented into binary values ​​using a global cost minimization technique (e.g., classifying each grid cell as potentially occupied or free space), and closed 2D contours represented in the resulting binary segmentation map can be extracted using any known technique.

[0084] Thus, navigable space estimation 460 can identify a corresponding track of the data collection vehicle, sample a point associated with the track (e.g., a point on the track, such as the center point of the track), identify a parent contour of the extracted closed 2D contour that encloses the sampled point, and identify the parent contour as the navigable space boundary 465. Additionally or alternatively, the parent contour can be assigned a corresponding ground height value (e.g., from ground model 435) to generate a corresponding 3D contour, which can be used as the navigable space boundary 465. In some embodiments, the 3D contour is projected into the corresponding 2D view to generate a 2D boundary of a ground truth segmentation mask, either of which can be used as the navigable space boundary 465. In some embodiments, the extracted child contours of the parent contour can be assumed to represent static hazards, assigned corresponding heights, projected into the ground truth segmentation mask, and used to carve out regions from the ground truth navigable space and / or the corresponding navigable space boundary 465. These are just a few examples, and other implementations are possible. In an example embodiment, the navigable space boundary 465 is represented using a timestamped closed 3D polygon representing the navigable space associated with the corresponding frame of the LiDAR data 402 or a global closed 3D polygon representing the navigable space for a set of frames of the LiDAR data 402. Example uses of the navigable space boundary 465 include planning and controlling the ego-machine during the operational phase (e.g., planning and executing a safe path in a real-world scenario) and geometric pre-filtering of labels to restrict ground truth data to elements that lie within the navigable space (e.g., during training). Figure 1 When running the object detection pipeline 100, random transformer queries that lie outside the ground-truth navigable space are omitted). Figure 7 is an illustration of an example boundary 710 of a navigable space in accordance with certain embodiments of the present disclosure.

[0085] In some embodiments, static obstacle detection 470 may use extracted sub-contours of navigable space boundary 465 (e.g., detected contours within the navigable space that are not part of the navigable space) to identify ground truth hazardous objects 475. Depending on the use case and / or embodiment, extracted sub-contours that are longer than a specified length and / or represent or enclose LiDAR points below ground model 435 may be filtered out. The (e.g., remaining) sub-contours may be assumed to represent static obstacles, and 2D and / or 3D bounding boxes or other bounding shapes may be generated (e.g., using the maximum and minimum heights of the LiDAR points enclosed by each contour) and may be used as ground truth hazardous objects 475. In example embodiments, hazardous objects 475 may be represented using a 3D bounding box or other bounding shape that describes the extent of each hazardous object and / or using some or all of the points contained in the bounding shape. Example uses of hazardous objects 475 include online static obstacle estimation (e.g., training Figure 1 The object detection pipeline 100 in

[10] is used to detect static hazards (such as road debris) and an online weather classification algorithm (e.g., providing annotated environmental data that highlights how obstacle appearance changes under different weather conditions). Figure 8 is an illustration of an example navigable space 810 and example extracted road hazards 820 in accordance with some embodiments of the present disclosure.

[0086] In some embodiments, points belonging to detected dynamic obstacles 420 are filtered out and the remaining non-static scene points 480 in the scene are identified. Non-static scene points 480 may belong to weather or road conditions, such as rain or snow in the air or on the ground. Therefore, in some embodiments, driving condition detection 482 can be applied to support the weather condition detection task. For example, driving condition detection 482 can sample a set of non-static scene points 480 from an area that is unlikely to contain any obstacles, such as a cube (e.g., one meter in length) or other volume in front of the data collection vehicle and above the ground model 435. The non-static scene points 480 detected in this area can be accumulated (e.g., over a specified duration or number of frames), and the power distribution of the resulting non-static scene points 480 across different frequency components can be quantified (e.g., using power spectrum analysis). Thus, the power level (or change in power level) at a specified frequency can be assigned a label representing the corresponding weather condition type (e.g., rain, snow, fog, dust, clear). Additionally or alternatively, corresponding levels of visibility impairment may be classified (e.g., using a linear relationship between the number and / or power of non-static scene points 480 and corresponding visibility impairment classes). To support the driving condition detection task, driving condition detection 482 may use a similar process to identify and / or accumulate non-static scene points 480 from areas on the ground model 435 (e.g., in front of the data collection vehicle), assign labels representing corresponding types of surface conditions (e.g., wet, snowy, icy, damp, dry), and / or assign labels representing corresponding degrees of surface damage.

[0087] Weather and / or surface condition labels 485 may be represented using one or more labels (e.g., per point, point scene) representing a ground truth class of a weather condition (e.g., rain, snow, fog, dust, clear), a value (e.g., integer) (e.g., from 0 to 4) quantifying visibility impairment caused by weather, a ground truth class of a surface condition (e.g., wet, snow, icy, damp, dry), a value (e.g., integer) (e.g., from 0 to 4) quantifying surface damage (e.g., ground) caused by weather or road conditions, and / or other uses. Example uses of weather and / or surface condition labels 485 include training online weather and surface (e.g., ground, road, pavement) condition detection or classification algorithms (e.g., such as Figure 3

[0014] The present invention also includes surface feature detection networks (described in some embodiments of the present invention), detecting when the autonomous vehicle encounters situations or conditions that are outside the predefined parameters it is designed to handle (its operational design domain or ODD), and data mining for specific weather scenarios.

[0088] In some embodiments, artifact detection 490 can identify non-static scene points 480 as artifacts. In some embodiments (e.g., where points identified as part of dynamic obstacles 420 were not previously filtered out), artifact detection 490 can subtract from non-static scene points 480 those points belonging to dynamic obstacles 420 and label the remaining points as artifacts. Each-point artifact label 495 can be represented using a per-point binary indicator (e.g., flag) assigned to each point in the dataset, indicating whether the point belongs to a non-solid, transient artifact (e.g., snowflakes, raindrops, dust particles, exhaust or vent plume condensate, etc.) that the ego machine can navigate through, or a solid surface (e.g., belonging to a car, pedestrian, traffic sign, tree, etc.) that must be avoided. Example uses of each-point artifact label 495 include training artifact detection or segmentation algorithms and improving LiDAR simulation of ephemeris.

[0089] Reference is now made to Figures 9 to 12 Each block of the methods 900-1200 described herein comprises a computational process that can be performed using any combination of hardware, firmware, and / or software. For instance, various functions can be carried out by a processor executing instructions stored in memory. The methods can also be embodied as computer-usable instructions stored on computer storage media. The methods can be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. In addition, the methods can be described in the context of the object detection pipeline 100, Figure 1 the surface feature detection pipeline 300, or Figure 3 the ground truth data generation pipeline 400. However, the methods can additionally or alternatively be performed by any one system or any combination of systems, including but not limited to the systems described herein. Figure 4

[0090] Figure 9 is a flowchart illustrating a method 900 for hazard detection, in accordance with some embodiments of the present disclosure. The method 900 includes detecting, at block B902, one or more hazards in an environment of an ego machine based at least on processing representations of image data and LiDAR data corresponding to the environment of the ego machine by one or more neural networks (NNs) comprising one or more transformers. For example, in the embodiment shown in Figure 1 the object detection pipeline 100 uses image data 101 and LiDAR data 121 (e.g., using an ego machine such as 13A to 13D ​The query generator 140 may generate a set of object queries 145 including a set of reference or anchor 3D positions using candidate bounding shapes predicted by the auxiliary head 115 from the extracted image features 110 and / or candidate bounding shapes predicted by the auxiliary head 135 from the extracted LiDAR features 130, using randomly initialized 3D positions, and / or using (e.g., the top K) self-motion compensated converter predictions from a previous frame (e.g., for objects predicted with a threshold confidence level). A feature sampler 150 may sample image features 110 (e.g., in stereo views) and / or LiDAR features 130 (e.g., in top-down or bird's-eye views) around each 3D reference point at keypoint locations identified by projecting learned (e.g., deformable) and / or specified 2D or 3D offsets into corresponding feature maps, and a transformer (e.g., comprising one or more input layers 165, a transformer decoder 170, and one or more output heads 175) may use the sampled image and LiDAR features to extract a representation of whether an object (e.g., a hazard) is present at the 3D location corresponding to each object query 145, its shape, and / or its class.

[0091] Method 900 includes, at block B904, controlling one or more operations of the self-machine based at least on one or more hazards. Figure 1 In the object detection pipeline 100 of the present invention, the output head 175 of the converter can output a predicted position, size, and / or class representation of a detected object in a 3D environment. The object detection can be used by one or more control components of the autonomous vehicle, such as the controller 1336, the ADAS system 1338, and / or an autonomous driving software stack (e.g., the software stack described in U.S. Patent Application Publication No. 20210026355A1) executed on one or more components of the vehicle 1300 (e.g., the SoC 1304, the CPU 1318, the GPU 1320, etc.). For example, the parameters predicted by the converter can be decoded to generate a 3D bounding box or other bounding shape 180 and a corresponding class label and confidence score. These object detections can be provided to the control component, and the control component can use any known technique to use the object detection to navigate, plan, or otherwise perform one or more operations (e.g., obstacle avoidance, lane keeping, lane changing, merging, lane splitting, etc.) within the environment.

[0092] Figure 10is a flow chart illustrating a method 1000 for generating a ground truth representation of one or more static hazards according to some embodiments of the present disclosure. The method 1000 includes, at block B1002, detecting one or more static scene points represented by one or more LiDAR detections. For example, regarding Figure 4 In the ground truth data generation pipeline 400, point filtering 450 can be applied to label detected LiDAR points as static or non-static based on the consistency of multiple observations. In some embodiments, space carving can be applied in 2D or 3D to iteratively carve out areas of space that are unlikely to contain static objects (and corresponding static points). For example, consistency measures of presence and / or distance can be quantified and used to derive a consistency measure for each LiDAR point, and points with a consistency measure below a specified threshold can be classified as non-static (e.g., a non-static part of a scene), while points with a consistency measure above the threshold can be classified as static (e.g., part of a static object or a static part of a scene).

[0093] Method 1000 includes, at block B1004, detecting one or more static hazards represented by one or more static scene points. Figure 4 In the ground truth data generation pipeline 400, as part of navigable space estimation 460 or static obstacle detection 470, static scene points 455 can be projected onto a grid representation of a ground model 435, aggregated per cell, and compensated for noise, and an occupancy grid can be generated by evaluating the resulting height above ground and the corresponding estimated surface curvature for each cell using any known height- and curvature-based occupancy scoring function. The resulting occupancy scores can be segmented into binary values ​​using a global cost minimization technique (e.g., classifying each grid cell as potentially occupied or free space), and closed 2D contours represented in the resulting binary segmentation map can be extracted using any known technique. The parent contour of the extracted closed 2D contour that encloses one or more points on the ego machine's trajectory can be identified as the navigable space boundary 465, and the extracted sub-contours of the navigable space boundary 465 can be used to identify ground truth hazardous objects 475. Depending on the use case and / or embodiment, extracted sub-contours that exceed a specified length and / or represent or enclose LiDAR points below the ground model 435 can be filtered out. It may be assumed that the (eg, remaining) sub-contours represent static obstacles.

[0094] Method 1000 includes, at block B1006, generating a truth representation of one or more static hazards. Figure 4In the ground truth data generation pipeline 400, 2D and / or 3D bounding boxes or other bounding shapes (e.g., using the maximum and minimum heights of the LiDAR points enclosed by each contour) can be generated and used as ground truth hazardous objects 475. In an example embodiment, hazardous objects 475 can be represented using a 3D bounding box or other bounding shape that describes the extent of each hazardous object and / or using some or all of the points contained in the bounding shape.

[0095] Figure 11 is a flow chart illustrating a method 1100 for surface feature detection according to some embodiments of the present disclosure. The method 1100 includes, at block B 1102, processing image data and a representation of LiDAR data corresponding to the environment of the ego machine based at least on one or more neural networks (NNs) including one or more transducers to detect one or more features of a surface in the environment. For example, the surface feature detection pipeline 300 can be used to detect one or more features of a road or other surface in the environment. More specifically, the query generator 340 can generate a set of surface queries 345 using one or more ego machine tracks (e.g., one or more tire tracks) predicted by the path generator 350 based on corresponding wheel angles 337 and / or ego motion compensated transducer predictions from a previous frame (e.g., for surface positions predicted with a threshold confidence level). Thus, the transformer formed by one or more input layers 365, transformer decoders 370, and one or more output heads 375 can use the sampled images and LiDAR features to extract representations of surface features 380 (e.g., height, distance, driving conditions, etc.) at 3D locations corresponding to each surface query, and can iteratively refine surface queries 345 representing candidate 3D locations on the surface to detect one or more curves of the surface.

[0096] Method 1100 includes, at block B1104, controlling one or more operations of the self-machine based at least on one or more features of the surface. Figure 3In the surface feature detection pipeline 300 of the autonomous vehicle, the output head 375 of the converter can output a representation of the predicted surface features 380 in the 3D environment, which can be used by one or more control components of the autonomous vehicle, such as the controller 1336, the ADAS system 1338, and / or an autonomous driving software stack (such as the software stack described in U.S. Patent Application Publication No. 20210026355A1) executed on one or more components of the vehicle 1300 (e.g., the SoC 1304, the CPU 1318, the GPU 1320, etc.). For example, the parameters predicted by the converter can be decoded to extract the height and / or uncertainty value of each sample point on the surface, the height and / or uncertainty value can be provided to the control component, and the control component can use any known technique to use the height and / or uncertainty value to navigate, plan, or otherwise perform one or more operations within the environment (e.g., obstacle or bump avoidance, lane keeping, lane changing, merging, lane splitting, adjusting the suspension system of the ego object or ego actor to match the current road surface, applying advance acceleration or deceleration based on the approaching road surface slope, mapping, etc.).

[0097] Figure 12 is a flow chart illustrating a method 1200 for generating a ground truth representation of one or more features of a detected ground surface according to some embodiments of the present disclosure. The method 1200 includes, at block B1202, generating a representation of the detected ground surface based at least on one or more LiDAR detections of an ego machine. For example, with respect to Figure 4 In the ground truth data generation pipeline 400, ground estimation 430 can use any known technique to estimate a representation of a ground model 435 based on (e.g., accumulated, ego-motion compensated, filtered) LiDAR data (e.g., classifying points predicted to be on the ground and projecting the points onto a grid and smoothing the projected values, e.g., as described in U.S. Patent Application No. 17 / 992,569) and / or based on image data (e.g., using 3D reconstruction to generate an estimated representation of the ground). In some embodiments, ground estimation 430 can include iteratively refining the accuracy of ground model 435 by minimizing (or approximately minimizing) a cost function that penalizes deviations between measured and estimated values ​​(data items) and / or deviations between adjacent measured values.

[0098] Method 1200 includes, at block B1204, generating a representation of one or more sampling points based at least on one or more trajectory samples of the ego machine. Figure 4 With reference to the ground truth data generation pipeline 400 and employing an example embodiment of a surface feature detection network involving features of a specified portion of the predicted ground surface (e.g., a set of points sampled along one or more predicted trajectories), sampling 440 can be applied to sample any number of points from the specified portion of the ground surface model 435. Figure 6 Assuming that the portion of ground model 610 of interest is an area extending outward from the data collection vehicle along one or more known tracks (e.g., track 620), one or more point segments or regions (e.g., region 630) proximate to the one or more tracks on ground model 610 can be selected and sampled to obtain ground truth points. In some embodiments, selected points in the selected point segments or regions of ground model 610 can be iteratively refined (e.g., as described above) and used to sample any number of points using any suitable sampling technique.

[0099] Method 1200 includes, at block B1206, generating a ground truth representation of one or more features of the detected ground surface at one or more sampling points. Figure 4 In the ground truth data generation pipeline 400, feature encoding 442 can be used to encode features of the sample points that are relevant to the desired downstream task or use case. Surface features 445 can be represented using: a scalar indicating a corresponding feature for each sample point on the surface (e.g., height, distance, a quantitative representation of surface damage, such as those represented by surface condition labels 485, surface gradient or curvature, etc.), and / or a regularly spaced grid (e.g., spanning part or all of the scene) where one or more values ​​for each grid cell encode the value of the corresponding feature.

[0100] The systems and methods described herein can be used by or in conjunction with non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more adaptive driver assistance systems (ADAS)), manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, flying boats, ships, space shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, trains, underwater vehicles, remotely operated vehicles (such as drones), and / or other vehicle types, but are not limited to these. In addition, the systems and methods described herein can be used for a variety of purposes, such as, but not limited to, machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, safety and surveillance, simulation and digital twins, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or participant simulation and / or digital twins, data center processing, conversational AI, light transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation of 3D assets, cloud computing, generative AI, and / or any other suitable application.

[0101] For example, the present technology (e.g., Figure 1 Object detection pipeline 100, Figure 3 Surface feature detection pipeline 300, Figure 4The ground truth data generation pipeline 400, portions thereof, etc., can be used in various robotics implementations (e.g., navigating surfaces or objects in a warehouse by a robot), and can be used in applications such as NVIDIA DRIVE Sim TM In some embodiments, the present invention relates to a method for training or validating a neural network (e.g., using virtual sensors and / or ray tracing to generate simulated input data corresponding to one or more components described herein, using real and / or simulated input data to generate simulated ground truth data, using real and / or simulated data to train or validate a neural network (such as the neural network or classical machine learning model described herein), and / or other methods. In general, simulations can be used to create simulated data sets that replicate various real-world conditions (e.g., that may be difficult or dangerous to observe in the real world), and training or validating neural networks or classical machine learning models in simulations can expose the models to a range of (e.g., rare or dangerous) scenarios in a controlled and safe environment.

[0102] The disclosed embodiments may be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines), systems implemented using robots, aviation systems, medical systems, boating systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems including one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least in part in a data center, systems for performing conversational AI operations, systems implementing one or more language models (such as one or more large language models (LLMs), one or more visual language models (VLMs), one or more multimodal language models, etc.), systems for performing light transport simulations, systems for performing collaborative content creation for 3D assets, systems implemented at least in part using cloud computing resources, and / or other types of systems.

[0103] Example autonomous vehicle

[0104] Figure 13Ais an illustration of an example autonomous vehicle 1300 according to some embodiments of the present disclosure. Autonomous vehicle 1300 (alternatively referred to herein as "vehicle 1300") may include, but is not limited to, a passenger vehicle such as a car, a truck, a bus, an emergency vehicle, a shuttle, an electric or motorized bicycle, a motorcycle, a fire truck, a police car, an ambulance, a boat, an engineering vehicle, an underwater vehicle, a robotic vehicle, a drone, an airplane, a vehicle coupled to a trailer (e.g., a semi-trailer tractor-trailer for hauling cargo), and / or another type of vehicle (e.g., a vehicle that is unmanned and / or that can accommodate one or more passengers). Autonomous vehicles are generally described in terms of automation levels, as defined by the National Highway Traffic Safety Administration (NHTSA) of the U.S. Department of Transportation and the Society of Automotive Engineers (SAE) in the "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles" (Standard No. J3016-201806, issued on June 15, 2018, Standard No. J3016-201609, issued on September 30, 2016, and previous and future versions of such standards). Vehicle 1300 may operate at one or more of the autonomous driving levels 3 to 5. Vehicle 1300 may operate at one or more of the autonomous driving levels 1 to 5. For example, according to an embodiment, vehicle 1300 may operate at driver assistance (Level 1), partial automation (Level 2), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5). The term “autonomy” as used herein may include any and / or all types of autonomy of a vehicle 1300 or other machine, such as fully autonomous, highly autonomous, conditionally autonomous, partially autonomous, with assisted autonomy, semi-autonomous, primarily autonomous, or other designations.

[0105] Vehicle 1300 may include components such as a chassis, a body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components of the vehicle. Vehicle 1300 may include a propulsion system 1350, such as an internal combustion engine, a hybrid power plant, an all-electric engine, and / or another type of propulsion system. Propulsion system 1350 may be connected to a drive train of vehicle 1300, which may include a transmission, to allow propulsion of vehicle 1300. Propulsion system 1350 may be controlled in response to receiving a signal from throttle / accelerator 1352.

[0106] A steering system 1354, which may include a steering wheel, may be used to steer vehicle 1300 (e.g., along a desired path or route) when propulsion system 1350 is operating (e.g., while the vehicle is in motion). Steering system 1354 may receive signals from steering actuator 1356. For fully automated (Level 5) functionality, the steering wheel may be optional. Brake sensor system 1346 may be used to operate the vehicle brakes in response to signals received from brake actuator 1348 and / or brake sensors.

[0107] May include one or more system on chip (SoC) 1304 ( Figure 13C ) and / or one or more GPUs can provide signals (e.g., representing commands) to one or more components and / or systems of vehicle 1300. For example, one or more controllers can send signals to operate vehicle brakes via one or more brake actuators 1348, to operate steering system 1354 via one or more steering actuators 1356, and to operate propulsion system 1350 via one or more throttles / accelerators 1352. One or more controllers 1336 can include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operational commands (e.g., signals representing commands) to enable autonomous driving and / or assist a human driver in driving vehicle 1300. One or more controllers 1336 can include a first controller 1336 for autonomous driving functionality, a second controller 1336 for functional safety functionality, a third controller 1336 for artificial intelligence functionality (e.g., computer vision), a fourth controller 1336 for infotainment functionality, a fifth controller 1336 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 1336 may handle two or more of the above functions, two or more controllers 1336 may handle a single function, and / or any combination thereof.

[0108] One or more controllers 1336 may provide signals for controlling one or more components and / or systems of vehicle 1300 in response to sensor data (eg, sensor input) received from one or more sensors. Sensor data may be received from, for example and without limitation, global navigation satellite system (“GNSS”) sensors 1358 (e.g., global positioning system sensors), RADAR sensors 1360 , ultrasonic sensors 1362 , LiDAR sensors 1364 , inertial measurement unit (IMU) sensors 1366 (e.g., accelerometers, gyroscopes, magnetic compasses, magnetometers, etc.), microphones 1396 , stereo cameras 1368 , wide-angle cameras 1370 (e.g., fisheye cameras), infrared cameras 1372 , surround cameras 1374 (e.g., 360-degree cameras), long-range and / or mid-range cameras 1398 , speed sensors 1344 (e.g., for measuring the velocity of the vehicle 1300 ), vibration sensors 1342 , steering sensors 1340 , brake sensors (e.g., as part of a brake sensor system 1346 ), one or more occupant monitoring system (OMS) sensors 1301 (e.g., one or more interior cameras), and / or other sensor types.

[0109] One or more of the controllers 1336 may receive input (e.g., represented by input data) from the instrument cluster 1332 of the vehicle 1300 and provide output (e.g., represented by output data, display data, etc.) via a human machine interface (HMI) display 1334, an audible annunciator, a speaker, and / or via other components of the vehicle 1300. These outputs may include information such as vehicle speed, velocity, time, data (e.g., Figure 13C The HMI display 1334 may include information such as high definition ("HD") 1322 of the vehicle 1300), position data (e.g., the position of the vehicle 1300, for example), direction, the positions of other vehicles (e.g., an occupancy grid), information about objects and object states as sensed by the controller 1336, and the like. For example, the HMI display 1334 may display information about the presence of one or more objects (e.g., street signs, warning signs, traffic light changes, etc.) and / or information about driving maneuvers that the vehicle has made, is making, or will make (e.g., changing lanes now, leaving 34B in two miles, etc.).

[0110] The vehicle 1300 also includes a network interface 1324 that can communicate over one or more networks using one or more wireless antennas 1326 and / or a modem. For example, the network interface 1324 can be capable of communicating over Long Term Evolution ("LTE"), Wideband Code Division Multiple Access ("WCDMA"), Universal Mobile Telecommunications System ("UMTS"), Global System for Mobile Communications ("GSM"), IMT-CDMA Multi-Carrier ("CDMA2000"), and the like. The one or more wireless antennas 1326 can also enable communication between objects in the environment (e.g., vehicles, mobile devices, and the like) using one or more local area networks such as Bluetooth, Bluetooth Low Energy ("LE"), Z-Wave, ZigBee, and the like, and / or one or more low power wide area networks (LPWANs) such as LoRaWAN, SigFox, and the like.

[0111] Figure 13B For use according to some embodiments of the present disclosure Figure 13A 13. Example camera positions and fields of view for autonomous vehicle 1300. The cameras and respective fields of view are an example embodiment and are not intended to be limiting. For example, additional and / or alternative cameras may be included and / or located at different locations on vehicle 1300.

[0112] The camera type used for the camera may include, but is not limited to, a digital camera that may be suitable for use with components and / or systems of the vehicle 1300. The camera may operate at Automotive Safety Integrity Level (ASIL) B and / or at another ASIL. The camera type may have any image capture rate, such as 60 frames per second (fps), 120fps, 240fps, and the like, depending on the embodiment. The camera may be capable of using a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include a red-white-white-white (RCCC) color filter array, a red-white-white-blue (RCCB) color filter array, a red-blue-green-white (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, a clear pixel camera such as a camera with an RCCC, RCCB, and / or RBGC color filter array may be used in an effort to improve light sensitivity.

[0113] In some examples, one or more of the cameras can be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function monocular camera can be installed to provide functions including lane departure warning, traffic sign assist, and intelligent headlamp control. One or more of the cameras (e.g., all of the cameras) can simultaneously record and provide image data (e.g., video).

[0114] One or more of these cameras can be mounted in a mounting assembly such as a custom designed (three-dimensional (“3D”) printed) assembly to cut off stray light and reflections from within the car (e.g., reflections from the dashboard reflected in the windshield mirror) that can interfere with the image data capture capabilities of the cameras. With respect to a wing mirror mounting assembly, the wing mirror assembly can be custom 3D printed such that the camera mounting plate matches the shape of the wing mirror. In some examples, one or more cameras can be integrated into the wing mirror. For side view cameras, one or more cameras can also be integrated into the four pillars at each corner of the cab.

[0115] Cameras with fields of view that include the portion of the environment in front of the vehicle 1300 (e.g., front-facing cameras) can be used for surround view to help identify the forward path and obstacles, and to assist in providing information critical to generating an occupancy grid and / or determining a preferred vehicle path with the help of one or more controllers 1336 and / or control SoCs. Front-facing cameras can be used to perform many of the same ADAS functions as LiDAR, including emergency braking, pedestrian detection, and collision avoidance. Front-facing cameras can also be used for ADAS functions and systems including lane departure warning (“LDW”), adaptive cruise control (“ACC”), and / or other functions such as traffic sign recognition.

[0116] A wide variety of cameras can be used in a front-facing configuration, including, for example, monocular camera platforms including complementary metal-oxide-semiconductor (“CMOS”) color imagers. Another example can be a wide-angle camera 1370, which can be used to perceive objects (e.g., pedestrians, intersection traffic, or bicycles) entering the field of view from the periphery. Although Figure 13B Although only one wide-angle camera is illustrated in FIG. 13, there can be any number (including zero) of wide-angle cameras 1370 on the vehicle 1300. In addition, any number of long-range cameras 1398 (e.g., pairs of long-view stereo cameras) can be used for depth-based object detection, especially for objects for which a neural network has not been trained. Long-range cameras 1398 can also be used for object detection and classification and basic object tracking.

[0117] Any number of stereo cameras 1368 may also be included in the front configuration. In at least one embodiment, one or more stereo cameras 1368 may include an integrated control unit that includes a scalable processing unit that may provide a multi-core microprocessor and programmable logic (FPGA) with an integrated controller area network ("CAN") or Ethernet interface on a single chip. Such a unit can be used to generate a 3D image of the vehicle's environment, including distance estimates for all points in the image. Alternative stereo cameras 1368 may include a compact stereo vision sensor that may include two camera lenses (one on the left and one on the right) and an image processing chip that may measure the distance from the vehicle to a target object and use the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 1368 may be used in addition to or alternatively to those described herein.

[0118] Cameras with a field of view that includes portions of the environment to the sides of the vehicle 1300 (e.g., side-view cameras) can be used for surround viewing, providing information used to create and update occupancy grids and generate side impact collision warnings. For example, surround cameras 1374 (e.g., Figure 13B Four surround cameras 1374 (shown in FIG) can be placed on vehicle 1300. Surround cameras 1374 can include wide-angle camera 1370, fisheye camera, 360-degree camera, and / or the like. For example, four fisheye cameras can be placed on the front, rear, and sides of the vehicle. In an alternative arrangement, the vehicle can use three surround cameras 1374 (e.g., left, right, and rear), and can utilize one or more other cameras (e.g., a forward-facing camera) as a fourth surround-view camera.

[0119] A camera having a field of view that includes a portion of the environment behind the vehicle 1300 (e.g., a rearview camera) can be used to assist with parking, surround view, rear collision warning, and creating and updating an occupancy grid. A variety of cameras can be used, including but not limited to cameras that are also suitable as front-facing cameras as described herein (e.g., long-range and / or mid-range cameras 1398, stereo cameras 1368, infrared cameras 1372, etc.).

[0120] A camera (e.g., one or more OMS sensors 1301) having a field of view (which includes a portion of the interior environment within the cabin of the vehicle 1300) can be used as part of an occupant monitoring system (OMS), such as, but not limited to, a driver monitoring system (DMS). For example, an OMS sensor (e.g., (multiple) OMS sensor 1301) can be used (e.g., by (multiple) controllers 1336) to track the gaze direction, head posture, and / or blinking of an occupant and / or driver. This gaze information can be used to determine the occupant's or driver's attention level (e.g., to detect drowsiness, fatigue, and / or distraction), and / or to take responsive actions to prevent harm to the occupant or driver. In some embodiments, data from the OMS sensor can be used to enable gaze-controlled operations triggered by the driver and / or non-driver occupants, such as, but not limited to, adjusting cabin temperature and / or airflow, opening and closing windows, controlling cabin lighting, controlling the entertainment system, adjusting mirrors, adjusting seat position, and / or other operations. In some embodiments, the OMS may be used for applications such as determining when an object and / or occupant is left in the vehicle cabin (eg, by detecting the presence of an occupant after the driver exits the vehicle).

[0121] Figure 13C For use according to some embodiments of the present disclosure Figure 13A 13. Block diagram of an example system architecture for an example autonomous vehicle 1300. It should be understood that this arrangement and other arrangements described herein are set forth merely as examples. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely. Further, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in conjunction with other components, and in any appropriate combination and location. The various functions described herein as being performed by entities may be implemented by hardware, firmware, and / or software. For example, the various functions may be implemented by a processor executing instructions stored in a memory.

[0122] Figure 13CEach of the components, features, and systems of vehicle 1300 is illustrated as being connected via bus 1302. Bus 1302 may include a controller area network (CAN) data interface (alternatively, referred to herein as a "CAN bus"). CAN may be a network internal to vehicle 1300 that assists in controlling various features and functions of vehicle 1300, such as actuation of brakes, acceleration, braking, steering, windshield wipers, and the like. The CAN bus may be configured to have tens or even hundreds of nodes, each with its own unique identifier (e.g., CAN ID). The CAN bus may be read to find steering wheel angle, ground speed, engine revolutions per minute (RPM), button positions, and / or other vehicle status indicators. The CAN bus may be ASIL B compliant.

[0123] Although bus 1302 is described here as a CAN bus, this is not intended to be limiting. For example, FlexRay and / or Ethernet may be used in addition to or in lieu of a CAN bus. Furthermore, although bus 1302 is represented by a single line, this is not intended to be limiting. For example, there may be any number of buses 1302, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 1302 may be used to perform different functions and / or may be used for redundancy. For example, a first bus 1302 may be used for collision avoidance functionality, and a second bus 1302 may be used for drive control. In any example, each bus 1302 may communicate with any component of vehicle 1300, and two or more buses 1302 may communicate with the same component. In some examples, each SoC 1304 , each controller 1336 , and / or each computer within the vehicle may have access to the same input data (e.g., input from sensors of the vehicle 1300 ) and may be connected to a common bus such as a CAN bus.

[0124] The vehicle 1300 may include one or more controllers 1336, such as those described herein. Figure 13A Controller 1336 may be used for a variety of functions. Controller 1336 may be coupled to any of the other various components and systems of vehicle 1300 and may be used for control of vehicle 1300, artificial intelligence of vehicle 1300, infotainment for vehicle 1300, and / or the like.

[0125] The vehicle 1300 may include one or more system-on-chips (SoCs) 1304. The SoC 1304 may include a CPU 1306, a GPU 1308, a processor 1310, a cache 1312, an accelerator 1314, a data store 1316, and / or other components and features not shown. The SoC 1304 may be used to control the vehicle 1300 in a variety of platforms and systems. For example, one or more SoCs 1304 may be combined with an HD 1322 in a system (e.g., a system of the vehicle 1300), which may be connected to a server (e.g., a computer) via a network interface 1324. Figure 13D One or more servers 1378) obtain refreshes and / or updates.

[0126] CPU 1306 may include a CPU cluster or CPU complex (alternatively, referred to herein as "CCPLEX"). CPU 1306 may include multiple cores and / or L2 caches. For example, in some embodiments, CPU 1306 may include eight cores in a coherent multiprocessor configuration. In some embodiments, CPU 1306 may include four dual-core clusters, each with a dedicated L2 cache (e.g., a 2MB L2 cache). CPU 1306 (e.g., CCPLEX) may be configured to support simultaneous cluster operations, allowing any combination of CPU 1306 clusters to be active at any given time.

[0127] CPU 1306 may implement power management capabilities including one or more of the following features: each hardware block may be automatically clock gated when idle to conserve dynamic power; each core clock may be gated when the core is not actively executing instructions due to the execution of WFI / WFE instructions; each core may be independently power gated; each core cluster may be independently clock gated when all cores are clock gated or power gated; and / or each core cluster may be independently power gated when all cores are power gated. CPU 1306 may further implement an enhanced algorithm for managing power states, in which allowed power states and expected wakeup times are specified, and hardware / microcode determines the optimal power state to enter for the core, cluster, and CCPLEX. The processing core may support a simplified power state entry sequence in software, with this work being offloaded to the microcode.

[0128] GPU 1308 may include an integrated GPU (alternatively referred to herein as an "iGPU"). GPU 1308 may be programmable and efficient for parallel workloads. In some examples, GPU 1308 may use an enhanced tensor instruction set. GPU 1308 may include one or more streaming microprocessors, each of which may include an L1 cache (e.g., an L1 cache with at least 96KB of storage capacity), and two or more of these streaming microprocessors may share an L2 cache (e.g., an L2 cache with 512KB of storage capacity). In some embodiments, GPU 1308 may include at least eight streaming microprocessors. GPU 1308 may use a computing application programming interface (API). In addition, GPU 1308 may use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0129] In the case of automotive and embedded use, GPU 1308 can be power optimized to achieve optimal performance. For example, GPU 1308 can be manufactured on fin field effect transistors (FinFETs). However, this is not intended to be limiting, and GPU 1308 can be manufactured using other semiconductor manufacturing processes. Each streaming microprocessor can incorporate several mixed precision processing cores divided into multiple blocks. For example and without limitation, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed precision NVIDIA tensor cores for deep learning matrix arithmetic, L0 instruction cache, warp scheduler, dispatch unit and / or 64KB register file. In addition, the streaming microprocessor may include independent parallel integer and floating point data paths to provide efficient execution of workloads using a mix of computation and addressing calculations. The streaming microprocessor may include independent thread scheduling capabilities to allow finer-grained synchronization and cooperation between parallel threads. The streaming microprocessor may include a combined L1 data cache and shared memory unit to improve performance while simplifying programming.

[0130] GPU 1308 can include high bandwidth memory (HBM) and / or a 16GB HBM2 memory subsystem that provides a peak memory bandwidth of approximately 900 GB / s in some examples. In some examples, synchronous graphics random access memory (SGRAM), such as fifth generation graphics double data rate synchronous random access memory (GDDR5), can be used in addition to or in lieu of HBM memory.

[0131] GPU 1308 can include a unified memory technology that includes access counters to allow memory pages to be migrated more precisely to the processors that access them most frequently, improving efficiency of memory ranges shared between processors. In some examples, address translation services (ATS) support can be used to allow GPU 1308 to directly access CPU 1306 page tables. In such examples, when a GPU 1308 memory management unit (MMU) experiences a miss, an address translation request can be transmitted to CPU 1306. In response, CPU 1306 can look up a virtual-to-physical mapping for the address in its page tables and transmit the translation back to GPU 1308. In this way, the unified memory technology can allow a single unified virtual address space for memory of both CPU 1306 and GPU 1308, simplifying GPU 1308 programming and porting applications to GPU 1308.

[0132] In addition, GPU 1308 can include access counters that can track how frequently GPU 1308 accesses other processors’ memory. The access counters can help ensure that memory pages are migrated to the physical memory of the processor that accesses these pages most frequently.

[0133] SoC 1304 can include any number of caches 1312, including those described herein. For example, the caches 1312 can include an L3 cache that is available to both CPU 1306 and GPU 1308 (e.g., connected to both CPU 1306 and GPU 1308). The caches 1312 can include a write-back cache that can track the state of a line, for example, by using a cache coherency protocol (e.g., MEI, MESI, MSI, etc.). Depending on the embodiment, the L3 cache can include 4 MB or more, although smaller cache sizes can also be used.

[0134] SoC 1304 can include an arithmetic logic unit (ALU) that can be utilized in processing to perform any of a variety of tasks or operations with respect to vehicle 1300, such as processing a DNN. In addition, SoC 1304 can include a floating point unit (FPU) (or other mathematical co-processor or digital co-processor type) for performing mathematical operations within the system. For example, SoC 1304 can include one or more FPUs integrated as execution units within CPU 1306 and / or GPU 1308.

[0135] The SoC 1304 can include one or more accelerators 1314 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, the SoC 1304 can include a hardware accelerator cluster that can include optimized hardware accelerators and / or large on-chip memory. The large on-chip memory (e.g., 4MB SRAM) can allow the hardware accelerator cluster to accelerate neural networks and other computations. The hardware accelerator cluster can be used to supplement the GPU 1308 and offload some of the tasks of the GPU 1308 (e.g., freeing up more cycles of the GPU 1308 for performing other tasks). As one example, the accelerators 1314 can be used for targeted workloads (e.g., perception, convolutional neural networks (CNNs), etc.) that are stable enough to accelerate easily. As used herein, the term “CNN” can include all types of CNNs, including region-based or region with convolutional neural networks (RCNNs) and fast RCNNs (e.g., for object detection).

[0136] The accelerators 1314 (e.g., hardware accelerator cluster) can include a deep learning accelerator (DLA). The DLA can include one or more tensor processing units (TPUs) that can be configured to provide an additional 100 billion operations per second for deep learning applications and inferencing. The TPUs can be accelerators that are configured to perform and optimized for performing image processing functions (e.g., for CNNs, RCNNs, etc.). The DLA can be further optimized for a specific set of neural network types and floating point operations and inferencing. The design of the DLA can provide higher performance per mm than general purpose GPUs and far exceeds the performance of CPUs. The TPUs can perform several functions including single instance convolution functions, support INT8, INT16, and FP16 data types for both features and weights, for example, and post-processor functions.

[0137] The DLA can perform neural networks, especially CNNs, on processed or unprocessed data, quickly and efficiently for any of a wide variety of functions, such as and not limited to: CNNs for object recognition and detection using data from a camera sensor; CNNs for distance estimation using data from a camera sensor; CNNs for emergency vehicle detection and identification and detection using data from a microphone; CNNs for face recognition and vehicle owner identification using data from a camera sensor; and / or CNNs for safety and / or safety related events.

[0138] The DLA can perform any of the functions of the GPU 1308, and by using an inference accelerator, for example, a designer can target the DLA or GPU 1308 for any function. For example, a designer can focus the processing of CNNs and floating point operations on the DLA, and leave other functions to the GPU 1308 and / or other accelerators 1314.

[0139] The accelerators 1314 (e.g., a cluster of hardware accelerators) can include a programmable vision accelerator (PVA), which can be alternatively referred to herein as a computer vision accelerator. The PVA can be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. The PVA can provide a balance between performance and flexibility. For example, each PVA can include, for example and without limitation, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors.

[0140] The RISC cores can interact with image sensors (e.g., image sensors of any of the cameras described herein), image signal processors, and / or the like. Each of these RISC cores can include any number of memories. Depending on the embodiment, the RISC cores can use any of several protocols. In some examples, the RISC cores can execute a real-time operating system (RTOS). The RISC cores can be implemented using one or more integrated circuit devices, application specific integrated circuits (ASICs), and / or memory devices. For example, the RISC cores can include an instruction cache and / or a tightly coupled RAM.

[0141] The DMA can allow components of the PVA to access system memory independently of the CPU 1306. The DMA can support any number of features to provide optimizations to the PVA, including, but not limited to, supporting multi-dimensional addressing and / or circular addressing. In some examples, the DMA can support addressing up to six or more dimensions, which can include block width, block height, block depth, horizontal block stride, vertical block stride, and / or depth stride.

[0142] A vector processor can be a programmable processor that can be designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing capabilities. In some examples, the PVA can include a PVA core and two vector processing subsystem partitions. The PVA core can include a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem can operate as the main processing engine of the PVA and can include a vector processing unit (VPU), an instruction cache, and / or a vector memory (e.g., VMEM). The VPU core can include a digital signal processor, such as, for example, a single instruction multiple data (SIMD), a very long instruction word (VLIW) digital signal processor. The combination of SIMD and VLIW can enhance throughput and speed.

[0143] Each of the vector processors can include an instruction cache and can be coupled to dedicated memory. As a result, in some examples, each of the vector processors can be configured to execute independently of the other vector processors. In other examples, the vector processors included in a particular PVA can be configured to employ data parallelism. For example, in some embodiments, multiple vector processors included in a single PVA can execute the same computer vision algorithm, but on different regions of an image. In other examples, the vector processors included in a particular PVA can execute different computer vision algorithms simultaneously on the same image, or even execute different algorithms on sequential images or portions of images. Among other things, any number of PVAs can be included in a hardware accelerator cluster, and any number of vector processors can be included in each of these PVAs. In addition, the PVAs can include additional error correction code (ECC) memory to enhance overall system security.

[0144] The accelerator 1314 (e.g., a hardware accelerator cluster) may include an on-chip computer vision network and SRAM to provide high bandwidth, low latency SRAM for the accelerator 1314. In some examples, the on-chip memory may include at least 4MB of SRAM consisting of, for example and without limitation, eight field-configurable memory blocks that can be accessed by both the PVA and the DLA. Each pair of memory blocks may include an advanced peripheral bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory may be used. The PVA and DLA may access the memory via a backbone that provides high-speed memory access to the PVA and DLA. The backbone may include an on-chip computer vision network that interconnects the PVA and DLA to the memory (e.g., using APB).

[0145] The on-chip computer vision network can include an interface that ensures that both the PVA and DLA provide ready and valid signals before transmitting any control signals / addresses / data. Such an interface can provide separate phases and separate channels for transmitting control signals / addresses / data, as well as burst-based communication for continuous data transmission. This type of interface can comply with ISO 26262 or IEC 61508 standards, but other standards and protocols can also be used.

[0146] In some examples, SoC 1304 may include a real-time ray tracing hardware accelerator such as that described in U.S. patent application Ser. No. 16 / 101,232 filed on Aug. 10, 2018. The real-time ray tracing hardware accelerator may be used to quickly and efficiently determine the position and extent of objects (e.g., within a world model) to generate real-time visualization simulations for RADAR signal interpretation, for sound propagation synthesis and / or analysis, for SONAR system simulations, for general wave propagation simulations, for comparison with LiDAR data for positioning and / or other functions, and / or for other uses. In some embodiments, one or more tree traversal units (TTUs) may be used to perform one or more ray tracing related operations.

[0147] The accelerator 1314 (e.g., a hardware accelerator cluster) has a wide range of uses in autonomous driving. The PVA can be a programmable vision accelerator that can be used in key processing stages in ADAS and autonomous vehicles. The capabilities of the PVA are a good match for algorithmic domains that require predictable processing, low power, and low latency. Therefore, the PVA performs well on semi-intensive or intensive rule computations, and even on small data sets that require predictable runtimes with low latency and low power. Therefore, in the context of platforms for autonomous vehicles, the PVA is designed to run classic computer vision algorithms because they are efficient at object detection and integer math operations.

[0148] For example, according to one embodiment of the technology, PVA is used to perform computer stereo vision. In some examples, a semi-global matching-based algorithm can be used, but this is not intended to be limiting. Many applications for Level 3-5 autonomous driving require on-the-fly motion estimation / stereo matching (e.g., structure from motion, pedestrian recognition, lane detection, etc.). PVA can perform computer stereo vision functions on input from two monocular cameras.

[0149] In some examples, PVA can be used to perform dense optical flow, by processing raw RADAR data (e.g., using a 4D Fast Fourier Transform) to provide processed RADAR. In other examples, PVA is used for time-of-flight depth processing, by processing raw time-of-flight data to provide processed time-of-flight data.

[0150] DLA can be used to run any type of network to enhance control and driving safety, including, for example, a neural network that outputs a confidence measure for each object detection. Such confidence values ​​can be interpreted as probabilities, or as providing a relative "weight" of each detection compared to other detections. This confidence value enables the system to make further decisions about which detections should be considered true positive detections rather than false positive detections. For example, the system can set a threshold for the confidence level and only consider detections that exceed the threshold as true positive detections. In an automatic emergency braking (AEB) system, a false positive detection would cause the vehicle to automatically perform emergency braking, which is obviously undesirable. Therefore, only the most confident detections should be considered as triggers for AEB. DLA can run a neural network for regressing confidence values. The neural network may take as its input at least some subset of parameters, such as bounding box dimensions, a ground plane estimate obtained (e.g., from another subsystem), inertial measurement unit (IMU) sensor 1366 output related to the orientation and range of the vehicle 1300, 3D position estimates of objects obtained from the neural network and / or other sensors (e.g., LiDAR sensor 1364 or RADAR sensor 1360), etc.

[0151] SoC 1304 may include one or more data stores 1316 (e.g., memory). Data stores 1316 may be on-chip memory of SoC 1304 that may store neural networks to be executed on the GPU and / or DLA. In some examples, data stores 1316 may be large enough to store multiple instances of the neural network for redundancy and safety. Data stores 1316 may include L2 or L3 cache 1312. References to data stores 1316 may include references to memory associated with the PVA, DLA, and / or other accelerators 1314 as described herein.

[0152] SoC 1304 may include one or more processors 1310 (e.g., embedded processors). Processor 1310 may include a boot and power management processor, which may be a dedicated processor and subsystem for handling boot power and management functions and related safety implementations. The boot and power management processor may be part of the SoC 1304 boot sequence and may provide runtime power management services. The boot power and management processor may provide clock and voltage programming, assist system low-power state transitions, SoC 1304 thermal and temperature sensor management, and / or SoC 1304 power state management. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to temperature, and SoC 1304 may use the ring oscillator to detect the temperature of CPU 1306, GPU 1308, and / or accelerator 1314. If the temperature is determined to exceed a threshold, the boot and power management processor may enter a temperature fault routine and place SoC 1304 in a lower power state and / or place vehicle 1300 in a driver safety parking mode (e.g., to safely park vehicle 1300).

[0153] The processor 1310 may also include a set of embedded processors that can be used as an audio processing engine. The audio processing engine can be an audio subsystem that allows for full hardware support for multi-channel audio through multiple interfaces and a wide range of flexible audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core having a digital signal processor with dedicated RAM.

[0154] The processor 1310 may also include an always-on processor engine that can provide the necessary hardware features to support low-power sensor management and wake-up use cases. The always-on processor engine may include a processor core, tightly coupled RAM, supporting peripherals (such as timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0155] The processor 1310 may also include a safety cluster engine, which includes a dedicated processor subsystem that handles safety management of automotive applications. The safety cluster engine may include two or more processor cores, tightly coupled RAM, supporting peripherals (such as timers, interrupt controllers, etc.), and / or routing logic. In safety mode, the two or more cores can operate in lockstep mode and act as a single core with comparison logic to detect any differences between their operations.

[0156] Processor 1310 may also include a real-time camera engine, which may include a dedicated processor subsystem for handling real-time camera management.

[0157] Processor 1310 may also include a high dynamic range signal processor, which may include an image signal processor, which is a hardware engine that is part of the camera processing pipeline.

[0158] The processor 1310 may include a video image compositer, which may be a processing block (e.g., implemented on a microprocessor) that implements the video post-processing functions required by the video playback application to produce the final image for the player window. The video image compositer may perform lens distortion correction for the wide-angle camera 1370, the surround camera 1374, and / or for the in-cab monitoring camera sensor. The in-cab monitoring camera sensor is preferably monitored by a neural network running on another instance of the advanced SoC and is configured to recognize in-cab events and respond accordingly. The in-cab system may perform lip reading to activate mobile phone service and place calls, dictate emails, change vehicle destinations, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. Certain functions are only available to the driver when the vehicle is operating in autonomous mode and are disabled in other circumstances.

[0159] The video image compositer can include enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, in the presence of motion in the video, the noise reduction appropriately weights spatial information and downweights information provided by neighboring frames. In the case where an image or portion of an image does not include motion, the temporal noise reduction performed by the video image compositer can use information from previous images to reduce noise in the current image.

[0160] The video image compositor can also be configured to perform stereo rectification on the input stereo footage frames. The video image compositor can further be used for user interface composition when the operating system desktop is in use and the GPU 1308 does not need to continuously render new surfaces. Even when the GPU 1308 is powered on and active for 3D rendering, the video image compositor can be used to offload the GPU 1308 to improve performance and responsiveness.

[0161] The SoC 1304 may also include a Mobile Industry Processor Interface (MIPI) camera serial interface, a high-speed interface for receiving video and input from a camera, and / or a video input block that may be used for camera and related pixel input functions. The SoC 1304 may also include an input / output controller that may be controlled by software and may be used to receive I / O signals that are not assigned to a specific role.

[0162] The SoC 1304 can also include a wide range of peripheral device interfaces to allow communication with peripherals, audio codecs, power management, and / or other devices. The SoC 1304 can be used to process data from cameras (connected over Gigabit Multimedia Serial Link and Ethernet), sensors (e.g., LiDAR sensor 1364, RADAR sensor 1360, etc. that can be connected over Ethernet), data from the bus 1302 (e.g., speed of the vehicle 1300, steering wheel position, etc.), data from GNSS sensor 1358 (connected over Ethernet or CAN bus). The SoC 1304 can also include dedicated high-performance mass storage controllers, which can include their own DMA engines, and which can be used to free up the CPU 1306 from routine data management tasks.

[0163] The SoC 1304 can be an end-to-end platform with a flexible architecture that spans automation levels 3-5, providing an integrated functional safety architecture for a platform that leverages and efficiently uses computer vision and ADAS technology to achieve diversity and redundancy, along with deep learning tools. The SoC 1304 can be faster, more reliable, and even more energy and space efficient than conventional systems. For example, the accelerators 1314, when combined with the CPU 1306, GPU 1308, and data storage 1316, can provide a fast and efficient platform for level 3-5 autonomous vehicles.

[0164] The technology thus provides capabilities and functionality that cannot be achieved by conventional systems. For example, computer vision algorithms can be executed on CPUs that can be configured using high-level programming languages such as the C programming language to perform a wide variety of processing algorithms across a wide variety of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to, for example, execution time and power consumption. In particular, many CPUs cannot execute complex object detection algorithms in real time, which is a requirement for on-board ADAS applications and for practical level 3-5 autonomous vehicles.

[0165] In contrast to conventional systems, by providing a CPU complex, a GPU complex, and a cluster of hardware accelerators, the technology described herein allows multiple neural networks to be executed simultaneously and / or sequentially, and the results combined together to allow level 3-5 autonomous driving functionality. For example, a CNN executed on a DLA or dGPU (e.g., GPU 1320) can include text and word recognition, allowing a supercomputer to read and understand traffic signs, including signs for which a neural network has not been specifically trained. The DLA can also include a neural network that is able to recognize, interpret, and provide a semantic understanding of the sign, and pass that semantic understanding to a path planning module running on the CPU complex.

[0166] As another example, as required for Level 3, 4, or 5 driving, multiple neural networks can run simultaneously. For example, a warning sign consisting of "Caution: Flashing lights indicate icing conditions" along with a light can be interpreted by several neural networks, either independently or collectively. The sign itself can be identified as a traffic sign by a first neural network deployed (e.g., a trained neural network), and the text "Flashing lights indicate icing conditions" can be interpreted by a second neural network deployed, which informs the vehicle's path planning software (preferably executing on a CPU complex) that icing conditions exist when the flashing lights are detected. The flashing lights can be identified by operating a third neural network deployed over multiple frames, which informs the vehicle's path planning software of the presence (or absence) of the flashing lights. All three neural networks can run simultaneously, for example, within the DLA and / or on GPU 1308.

[0167] In some examples, a CNN for facial recognition and owner recognition can use data from a camera sensor to identify the presence of an authorized driver and / or owner of the vehicle 1300. The always-on sensor processing engine can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and in security mode, disable the vehicle when the owner leaves the vehicle. In this way, the SoC 1404 provides security against theft and / or carjacking.

[0168] In another example, a CNN for emergency vehicle detection and identification can use data from microphone 1396 to detect and identify emergency vehicle sirens. In contrast to conventional systems that use general classifiers to detect sirens and manually extract features, SoC 1304 uses CNNs to classify environmental and urban sounds as well as visual data. In a preferred embodiment, the CNN running on the DLA is trained to identify the relative closing speed of emergency vehicles (for example, by using the Doppler effect). The CNN can also be trained to identify emergency vehicles specific to the local area in which the vehicle is operating, as identified by the GNSS sensor 1358. Thus, for example, when operating in Europe, the CNN will seek to detect European sirens, and when in the United States, the CNN will seek to identify only North American sirens. Once an emergency vehicle is detected, with the assistance of the ultrasonic sensor 1362, the control program can be used to execute emergency vehicle safety routines, slowing the vehicle, pulling to the side of the road, stopping the vehicle, and / or idling the vehicle until the emergency vehicle passes.

[0169] The vehicle may include a CPU 1318 (e.g., a discrete CPU or dCPU) that may be coupled to the SoC 1304 via a high-speed interconnect (e.g., PCIe). The CPU 1318 may include, for example, an X86 processor. The CPU 1318 may be used to perform any of a variety of functions, including, for example, arbitrating potentially inconsistent results between ADAS sensors and the SoC 1304, and / or monitoring the status and health of the controller 1336 and / or the infotainment SoC 1330.

[0170] The vehicle 1300 may include a GPU 1320 (e.g., a discrete GPU or dGPU) that may be coupled to the SoC 1304 via a high-speed interconnect (e.g., NVIDIA's NVLINK). The GPU 1320 may provide additional artificial intelligence functionality, for example, by executing redundant and / or different neural networks, and may be used to train and / or update the neural network based at least in part on input from sensors of the vehicle 1300 (e.g., sensor data).

[0171] The vehicle 1300 may also include a network interface 1324, which may include one or more wireless antennas 1326 (e.g., one or more wireless antennas for different communication protocols, such as a cellular antenna, a Bluetooth antenna, etc.). The network interface 1324 may be used to allow wireless connections to the cloud (e.g., to a server 1378 and / or other network devices), to other vehicles, and / or to computing devices (e.g., a passenger's client device) via the Internet. To communicate with other vehicles, a direct link may be established between the two vehicles, and / or an indirect link may be established (e.g., across a network and through the Internet). The direct link may be provided using a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link may provide the vehicle 1300 with information about vehicles approaching the vehicle 1300 (e.g., vehicles in front of, to the sides of, and / or behind the vehicle 1300). This functionality may be part of the cooperative adaptive cruise control functionality of the vehicle 1300.

[0172] The network interface 1324 may include a SoC that provides modulation and demodulation functionality and enables the controller 1336 to communicate over a wireless network. The network interface 1324 may include an RF front-end for up-conversion from baseband to RF and down-conversion from RF to baseband. The frequency conversion may be performed by well-known processes and / or may be performed using a super-heterodyne process. In some examples, the RF front-end functionality may be provided by a separate chip. The network interface may include wireless functionality for communicating over LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0173] The vehicle 1300 may also include data storage 1328, which may include off-chip storage (e.g., outside the SoC 1304). The data storage 1328 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash memory, a hard disk, and / or other components and / or devices that can store at least one bit of data.

[0174] The vehicle 1300 may also include a GNSS sensor 1358. The GNSS sensor 1358 (e.g., GPS, assisted GPS sensor, differential GPS (DGPS) sensor, etc.) is used to assist with mapping, perception, occupancy grid generation, and / or path planning functions. Any number of GNSS sensors 1358 may be used, including, for example and without limitation, GPS using a USB connector with an Ethernet to serial (RS-232) bridge.

[0175] The vehicle 1300 may also include a RADAR sensor 1360. The RADAR sensor 1360 can be used by the vehicle 1300 for remote vehicle detection even in darkness and / or in adverse weather conditions. The RADAR functional safety level can be ASIL B. The RADAR sensor 1360 can use CAN and / or bus 1302 (e.g., to transmit data generated using the RADAR sensor 1360) for control and access to object tracking data, and in some examples access Ethernet to access raw data. A variety of RADAR sensor types can be used. For example and without limitation, the RADAR sensor 1360 can be suitable for front, rear, and side RADAR use. In some examples, a pulsed Doppler RADAR sensor is used.

[0176] The RADAR sensor 1360 can include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range side coverage, and so on. In some examples, long-range RADAR can be used for adaptive cruise control functions. The long-range RADAR system can provide a wide field of view (e.g., within a range of 250m) achieved through two or more independent scans. The RADAR sensor 1360 can help distinguish between static objects and moving objects and can be used by the ADAS system for emergency braking assistance and forward collision warning. The long-range RADAR sensor can include a single-station multimode RADAR with multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In the example with six antennas, the central four antennas can create a focused beam pattern that is designed to record the surroundings of the vehicle 1300 at a higher rate with minimal traffic interference from adjacent lanes. The other two antennas can expand the field of view, making it possible to quickly detect vehicles entering or leaving the lane of the vehicle 1300.

[0177] As an example, a medium-range RADAR system may include a range of up to 1360m (front) or 80m (rear) and a field of view of up to 42 degrees (front) or 1350 degrees (rear). A short-range RADAR system may include, but is not limited to, a RADAR sensor designed to be mounted on both ends of the rear bumper. When mounted on both ends of the rear bumper, such a RADAR sensor system can create two beams that continuously monitor the blind spots behind and beside the vehicle.

[0178] Short-range RADAR systems can be used in ADAS systems for blind spot detection and / or lane change assistance.

[0179] Vehicle 1300 may also include ultrasonic sensors 1362. Ultrasonic sensors 1362, which may be located on the front, rear, and / or sides of vehicle 1300, may be used for parking assistance and / or for creating and updating an occupancy grid. A variety of ultrasonic sensors 1362 may be used, and different ultrasonic sensors 1362 may have different detection ranges (e.g., 2.5 m, 4 m). Ultrasonic sensors 1362 may operate at functional safety level ASIL B.

[0180] Vehicle 1300 may include a LiDAR sensor 1364. LiDAR sensor 1364 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. LiDAR sensor 1364 may be ASIL B functional safety level. In some examples, vehicle 1300 may include multiple LiDAR sensors 1364 (e.g., two, four, six, etc.) that may use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0181] In some examples, the LiDAR sensor 1364 may be capable of providing a list of objects and their distances for a 360-degree field of view. Commercially available LiDAR sensors 1364 may have, for example, an advertised range of approximately 1300 meters, an accuracy of 2-3 cm, and support for 1300 Mbps Ethernet connections. In some examples, one or more non-obtrusive LiDAR sensors 1364 may be used. In such examples, the LiDAR sensor 1364 may be implemented as a small device that can be embedded in the front, back, sides, and / or corners of the vehicle 1300. In such examples, the LiDAR sensor 1364 may provide a field of view of up to 120 degrees horizontally and 35 degrees vertically, with a range of 200 meters, even for low-reflectivity objects. The front-mounted LiDAR sensor 1364 may be configured for a horizontal field of view between 45 and 135 degrees. Figure 13BExample long-range and short-range horizontal fields of view are shown for a LiDAR sensor 1364 having an example mounting position above the windshield, but other configurations, such as including a LiDAR sensor 1364 mounted on a grille (e.g., as Figure 13A Configurations with a LiDAR scanner mounted on a roof (e.g., for a data collection vehicle) are also possible.

[0182] In some examples, LiDAR technologies such as 3D flash LiDAR may also be used. 3D flash LiDAR uses flashes of laser as an emission source to illuminate the vehicle's surroundings up to about 200 m. The flash LiDAR unit includes a receiver that records the laser pulse transmission time and the reflected light on each pixel, which in turn corresponds to the range from the vehicle to the object. Flash LiDAR can allow a highly accurate and distortion-free image of the surrounding environment to be generated with each laser flash. In some examples, four flash LiDAR sensors can be deployed, one on each side of the vehicle 1300. Available 3D flash LiDAR systems include solid-state 3D staring array LiDAR cameras (e.g., non-scanning LiDAR devices) with no moving parts other than a fan. The flash LiDAR device can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture the reflected laser light in the form of a 3D range point cloud and co-registered intensity data. By using flash LiDAR, and because flash LiDAR is a solid-state device with no moving parts, the LiDAR sensor 1364 may be less susceptible to motion blur, vibration, and / or shock.

[0183] The vehicle may also include an IMU sensor 1366. In some examples, the IMU sensor 1366 may be located at the center of the rear axle of the vehicle 1300. The IMU sensor 1366 may include, for example and without limitation, an accelerometer, a magnetometer, a gyroscope, a magnetic compass, and / or other sensor types. In some examples, such as in a six-axis application, the IMU sensor 1366 may include an accelerometer and a gyroscope, while in a nine-axis application, the IMU sensor 1366 may include an accelerometer, a gyroscope, and a magnetometer.

[0184] In some embodiments, the IMU sensor 1366 can be implemented as a miniature, high-performance GPS-assisted inertial navigation system (GPS / INS) that combines micro-electromechanical systems (MEMS) inertial sensors, a high-sensitivity GPS receiver, and advanced Kalman filtering algorithms to provide estimates of position, velocity, and attitude. Thus, in some examples, the IMU sensor 1366 can allow the vehicle 1300 to estimate heading without requiring input from a magnetic sensor by directly observing and correlating velocity changes from the GPS to the IMU sensor 1366. In some examples, the IMU sensor 1366 and the GNSS sensor 1358 can be combined into a single integrated unit.

[0185] The vehicle may include microphones 1396 positioned in and / or around the vehicle 1300. The microphones 1396 may be used for, among other things, emergency vehicle detection and identification.

[0186] The vehicle may also include any number of camera types, including stereo cameras 1368, wide angle cameras 1370, infrared cameras 1372, surround cameras 1374, long and / or medium range cameras 1398, and / or other camera types. These cameras may be used to capture image data around the entire periphery of the vehicle 1300. The type of camera used depends on the embodiment and the requirements of the vehicle 1300, and any combination of camera types may be used to provide the necessary coverage around the vehicle 1300. Additionally, the number of cameras may vary depending on the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. As an example and not limitation, the cameras may support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet. Each of the cameras described herein may include a GMSL and / or Gigabit Ethernet network. Figure 13A and Figure 13B Described in more detail.

[0187] Vehicle 1300 may also include a vibration sensor 1342. Vibration sensor 1342 can measure vibrations of vehicle components, such as axles. For example, changes in vibration can indicate changes in the road surface. In another example, when two or more vibration sensors 1342 are used, the difference between the vibrations can be used to determine friction or slippage of the road surface (e.g., when there is a vibration difference between a powered drive shaft and a freely rotating shaft).

[0188] The vehicle 1300 may include an ADAS system 1338. In some examples, the ADAS system 1338 may include a SoC. The ADAS system 1338 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keeping assist (LKA), blind spot warning (BSW), rear cross traffic warning (RCTW), collision warning system (CWS), lane centering (LC), and / or other features and functions.

[0189] The ACC system can utilize RADAR sensors 1360, LiDAR sensors 1364, and / or cameras. The ACC system can include longitudinal ACC and / or lateral ACC. Longitudinal ACC monitors and controls the distance to the vehicle immediately in front of vehicle 1300, automatically adjusting the vehicle speed to maintain a safe distance from the vehicle in front. Lateral ACC maintains distance and, when necessary, recommends that vehicle 1300 change lanes. Lateral ACC is related to other ADAS applications such as LCA and CWS.

[0190] CACC uses information from other vehicles, which can be received from other vehicles indirectly via a wireless link via the network interface 1324 and / or the wireless antenna 1326 or via a network connection (e.g., via the Internet). A direct link can be provided by a vehicle-to-vehicle (V2V) communication link, while an indirect link can be an infrastructure-to-vehicle (I2V) communication link. Typically, the V2V communication concept provides information about the vehicle immediately ahead (e.g., the vehicle immediately ahead of the vehicle 1300 and in the same lane as it), while the I2V communication concept provides information about traffic further ahead. The CACC system can include either or both of the I2V and V2V information sources. Given information about the vehicle ahead of the vehicle 1300, CACC can be more reliable, and it has the potential to improve the smoothness of traffic flow and reduce road congestion.

[0191] The FCW system is designed to alert the driver to hazards so that the driver can take corrective action. The FCW system uses a front-facing camera and / or RADAR sensor 1360 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating component. The FCW system can provide warnings in the form of, for example, audible, visual warnings, vibrations, and / or rapid brake pulses.

[0192] The AEB system detects an impending forward collision with another vehicle or other object and can automatically apply the brakes if the driver does not take corrective action within specified time or distance parameters. The AEB system can use a front-facing camera and / or RADAR sensor 1360 coupled to a dedicated processor, DSP, FPGA and / or ASIC. When the AEB system detects a hazard, it typically first alerts the driver to take corrective action to avoid the collision, and if the driver does not take corrective action, the AEB system can automatically apply the brakes in an effort to prevent or at least mitigate the effects of the predicted collision. The AEB system can include technologies such as dynamic brake support and / or collision approach braking.

[0193] The LDW system provides visual, audible, and / or tactile warnings, such as steering wheel or seat vibrations, to alert the driver when the vehicle 1300 crosses a lane marking. When the driver indicates an intention to leave the lane by activating a turn signal, the LDW system is deactivated. The LDW system may utilize a front-facing camera coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as a display, speaker, and / or vibration components.

[0194] The LKA system is a variation of the LDW system. If the vehicle 1300 begins to leave the lane, the LKA system provides steering input or braking to correct the vehicle 1300.

[0195] The BSW system detects and warns the driver of vehicles in the car's blind spot. The BSW system can provide visual, audible, and / or tactile alerts to indicate that merging or changing lanes is unsafe. The system can provide additional warnings when the driver uses a turn signal. The BSW system can use a rear-facing camera and / or RADAR sensor 1360 coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as a display, speaker, and / or vibration component.

[0196] The RCTW system can provide visual, audible, and / or tactile notifications when an object is detected outside the range of the rear-mounted camera while the vehicle 1300 is in reverse. Some RCTW systems include automatic emergency braking (AEB) to ensure that the vehicle brakes are applied to avoid a collision. The RCTW system can use one or more rear-mounted RADAR sensors 1360 coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as a display, speaker, and / or vibration component.

[0197] Conventional ADAS systems can be prone to false positive results, which can annoy and distract the driver, but typically are not catastrophic because the ADAS system alerts the driver and allows the driver to decide whether the safety condition is truly present and act accordingly. However, in autonomous vehicle 1300, in the case of conflicting results, the vehicle 1300 itself must decide whether to heed the results from the primary computer or the secondary computer (e.g., first controller 1336 or second controller 1336). For example, in some embodiments, the ADAS system 1338 can be a secondary and / or auxiliary computer for providing perception information to the backup computer plausibility module. The backup computer plausibility monitor can run redundant software on hardware components to detect faults in perception and dynamic driving tasks. The output from the ADAS system 1338 can be provided to the supervisory MCU. If the output from the primary computer and the secondary computer conflict, the supervisory MCU must determine how to reconcile the conflict to ensure safe operation.

[0198] In some examples, the primary computer can be configured to provide a confidence score to the supervisory MCU indicating the primary computer’s confidence in the selected result. If the confidence score exceeds a threshold, the supervisory MCU can follow the direction of the primary computer regardless of whether the secondary computer provides a conflicting or inconsistent result. In the case where the confidence score does not satisfy the threshold and in the case where the primary computer and the secondary computer indicate different results (e.g., conflict), the supervisory MCU can arbitrate between the computers to determine the appropriate result.

[0199] The supervisory MCU can be configured to run a neural network that is trained and configured to determine, based at least in part on the output from the primary computer and the secondary computer, conditions under which the secondary computer provides false alarms. Thus, the neural network in the supervisory MCU can learn when the output of the secondary computer can be trusted and when it cannot. For example, when the secondary computer is a RADAR-based FCW system, the neural network in the supervisory MCU can learn when the FCW system is identifying metal objects that are not in fact dangerous, such as a drain grate or manhole cover that triggers an alert. Similarly, when the secondary computer is a camera-based LDW system, the neural network in the supervisory MCU can learn to disregard the LDW when a cyclist or pedestrian is present and lane departure is in fact the safest strategy. In embodiments that include a neural network running on the supervisory MCU, the supervisory MCU can include at least one of a DLA or GPU suitable for running the neural network with associated memory. In preferred embodiments, the supervisory MCU can include and / or be included as a component of the SoC 1304.

[0200] In other examples, the ADAS system 1338 can include a secondary computer that performs ADAS functions using traditional computer vision rules. As such, the secondary computer can use classic computer vision rules (if-then) and the presence of a neural network in the supervising MCU can improve reliability, safety, and performance. For example, the diversified implementation and the intentional non-identity make the overall system more fault-tolerant, especially for faults caused by software (or software-hardware interface) functions. For example, if there is a software bug or error in the software running on the primary computer and the non-identical software code running on the secondary computer provides the same overall result, then the supervising MCU can be more confident that the overall result is correct and that the bug in the software or hardware on the primary computer did not cause a substantial error.

[0201] In some examples, the output of the ADAS system 1338 can be fed to the perception block of the primary computer and / or the dynamic driving task block of the primary computer. For example, if the ADAS system 1338 indicates a forward collision warning due to an object immediately ahead, then the perception block can use that information in identifying the object. In other examples, the secondary computer can have its own neural network that is trained and thus reduces the risk of false positives as described herein.

[0202] The vehicle 1300 may also include an infotainment SoC 1330 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system may not be an SoC and may include two or more separate components. The infotainment SoC 1330 may include a combination of hardware and software that can be used to provide audio (e.g., music, personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), phone (e.g., hands-free calling), network connectivity (e.g., LTE, WiFi, etc.), and / or information services (e.g., a navigation system, rear parking assistance, radio data system, vehicle-related information such as fuel level, total distance covered, brake fuel level, oil level, door open / closed, air filter information, etc.) to the vehicle 1300. For example, the infotainment SoC 1330 may include a radio, a disc player, a navigation system, a video player, USB and Bluetooth connectivity, an onboard computer, in-car entertainment, WiFi, steering wheel audio controls, hands-free voice controls, a head-up display (HUD), an HMI display 1334, a telematics device, a control panel (e.g., for controlling and / or interacting with various components, features, and / or systems), and / or other components. The infotainment SoC 1330 may further be used to provide information (e.g., visual and / or auditory) to a user of the vehicle, such as information from an ADAS system 1338, autonomous driving information such as planned vehicle maneuvers, trajectories, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0203] The infotainment SoC 1330 may include GPU functionality. The infotainment SoC 1330 may communicate with other devices, systems, and / or components of the vehicle 1300 via a bus 1302 (e.g., a CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 1330 may be coupled to a supervisory MCU so that in the event of a failure of a primary controller 1336 (e.g., a primary and / or backup computer of the vehicle 1300), the infotainment system's GPU may perform some self-driving functions. In such an example, the infotainment SoC 1330 may place the vehicle 1300 in a driver-safe parking mode as described herein.

[0204] The vehicle 1300 may also include an instrument cluster 1332 (e.g., a digital instrument panel, an electronic instrument cluster, a digital instrument panel, etc.). The instrument cluster 1332 may include a controller and / or a supercomputer (e.g., a separate controller or a supercomputer). The instrument cluster 1332 may include a set of instruments, such as a speedometer, fuel level, oil pressure, a tachometer, an odometer, a turn indicator, a gear position indicator, a seat belt warning light, a parking brake warning light, an engine check light, airbag (SRS) system information, lighting controls, safety system controls, navigation information, etc. In some examples, information may be displayed and / or shared between the infotainment SoC 1330 and the instrument cluster 1332. Thus, the instrument cluster 1332 may be included as part of the infotainment SoC 1330, or vice versa.

[0205] Figure 13D For cloud-based servers and Figure 13A 13. System diagram of communication between an example autonomous vehicle 1300. System 1376 may include a server 1378, a network 1390, and a vehicle including vehicle 1300. Server 1378 may include multiple GPUs 1384(A)-1384(H) (collectively referred to herein as GPUs 1384), PCIe switches 1382(A)-1382(D) (collectively referred to herein as PCIe switches 1382), and / or CPUs 1380(A)-1380(B) (collectively referred to herein as CPUs 1380). GPUs 1384, CPUs 1380, and PCIe switches may be interconnected with a high-speed interconnect such as, for example and without limitation, NVLink interface 1388 developed by NVIDIA and / or PCIe connection 1386. In some examples, GPUs 1384 are connected via NVLink and / or NVSwitch SoCs, and GPUs 1384 and PCIe switches 1382 are connected via PCIe interconnects. Although eight GPUs 1384, two CPUs 1380, and two PCIe switches are shown, this is not intended to be limiting. Depending on the embodiment, each of the servers 1378 may include any number of GPUs 1384, CPUs 1380, and / or PCIe switches. For example, each of the servers 1378 may include eight, sixteen, thirty-two, and / or more GPUs 1384.

[0206] Server 1378 can receive image data from a vehicle via network 1390, the image data representing images showing unexpected or changed road conditions, such as recently begun road construction. Server 1378 can transmit neural network 1392, updated neural network 1392, and / or information 1394, including information about traffic and road conditions, to the vehicle via network 1390. Updates to information 1394 can include updates to HD 1322, such as information about construction sites, potholes, curves, flooding, or other obstacles. In some examples, neural network 1392, updated neural network 1392, and / or information 1394 can be generated from new training and / or data received from any number of vehicles in the environment and / or based on experience from training performed at a data center (e.g., using server 1378 and / or other servers).

[0207] Server 1378 can be used to train a machine learning model (e.g., a neural network) based on training data. The training data can be generated using the vehicle and / or can be generated in simulation (e.g., using a game engine). In some examples, the training data is labeled (e.g., in cases where the neural network benefits from supervised learning) and / or undergoes other preprocessing, while in other examples, the training data is not labeled and / or preprocessed (e.g., in cases where the neural network does not require supervised learning). Training can be performed according to any one or more categories of machine learning techniques, including but not limited to the following categories: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, joint learning, transfer learning, feature learning (including principal component and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including alternative dictionary learning), rule-based machine learning, anomaly detection, and any variants or combinations thereof. Once the machine learning model is trained, the machine learning model can be used by the vehicle (e.g., transmitted to the vehicle via network 1390), and / or the machine learning model can be used by server 1378 to remotely monitor the vehicle.

[0208] In some examples, server 1378 can receive data from the vehicle and apply the data to the latest real-time neural network for real-time intelligent reasoning. Server 1378 can include a deep learning supercomputer and / or a dedicated AI computer powered by GPU 1384, such as the DGX and DGX Station machines developed by NVIDIA. However, in some examples, server 1378 can include the deep learning infrastructure of a data center using only CPU power.

[0209] The deep learning infrastructure of server 1378 may be capable of rapid real-time inference and may use this capability to assess and verify the health of the processors, software, and / or associated hardware in vehicle 1300. For example, the deep learning infrastructure may receive periodic updates from vehicle 1300, such as an image sequence and / or objects located in the image sequence that vehicle 1300 has located (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure may run its own neural networks to identify objects and compare them to the objects identified by vehicle 1300, and if the results do not match and the infrastructure concludes that the AI ​​in vehicle 1300 has malfunctioned, server 1378 may transmit a signal to vehicle 1300 instructing the vehicle's 1300 fail-safe computer to take control, notify passengers, and complete a safe parking maneuver.

[0210] For inference, server 1378 may include a GPU 1384 and one or more programmable inference accelerators (e.g., NVIDIA's TensorRT). The combination of GPU-powered servers and inference acceleration can enable real-time responses. In other examples, such as where performance is less important, CPU, FPGA, and other processor-powered servers can be used for inference.

[0211] Example computing device

[0212] Figure 14 14 is a block diagram of an example computing device 1400 suitable for implementing some embodiments of the present disclosure. Computing device 1400 may include an interconnect system 1402 that directly or indirectly couples the following devices: memory 1404, one or more central processing units (CPUs) 1406, one or more graphics processing units (GPUs) 1408, a communication interface 1410, input / output (I / O) ports 1412, I / O components 1414, a power supply 1416, one or more presentation components 1418 (e.g., display(s)), and one or more logic units 1420. In at least one embodiment, computing device(s) 1400 may include one or more virtual machines (VMs), and / or any of its components may include virtual components (e.g., virtual hardware components). For non-limiting examples, one or more of GPUs 1408 may include one or more vGPUs, one or more of CPUs 1406 may include one or more vCPUs, and / or one or more of logic units 1420 may include one or more virtual logic units. As such, computing device(s) 1400 may include discrete components (e.g., a full GPU dedicated to computing device 1400), virtual components (e.g., a portion of a GPU dedicated to computing device 1400), or a combination thereof.

[0213] although Figure 14 The various blocks of are shown as being connected via interconnect system 1402 using wires, but this is not intended to be limiting and is provided for clarity only. For example, in some embodiments, presentation component 1418 (such as a display device) may be considered to be I / O component 1414 (e.g., if the display is a touch screen). As another example, CPU 1406 and / or GPU 1408 may include memory (e.g., memory 1404 may represent a storage device in addition to the memory of GPU 1408, CPU 1406, and / or other components). Thus, Figure 14 The computing devices are illustrative only. No distinction is made between such categories as "workstation," "server," "laptop," "desktop," "tablet," "client device," "mobile device," "handheld device," "game console," "electronic control unit (ECU)," "virtual reality system," and / or other device or system types, as all are considered Figure 14 within the range of computing devices.

[0214] Interconnect system 1402 can represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. Interconnect system 1402 can include one or more bus or link types, such as an Industry Standard Architecture (ISA) bus, an Extended Industry Standard Architecture (EISA) bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe) bus, and / or another type of bus or link. In some embodiments, there is a direct connection between components. As an example, CPU 1406 can be directly connected to memory 1404. Further, CPU 1406 can be directly connected to GPU 1408. In the case where there is a direct or point-to-point connection between components, interconnect system 1402 can include a PCIe link to perform the connection. In these examples, the PCI bus does not need to be included in computing device 1400.

[0215] Memory 1404 may include any of a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by computing device 1400. Computer-readable media can include volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, computer-readable media can include computer storage media and communication media.

[0216] Computer storage media may include volatile and nonvolatile media and / or removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 1404 may store computer-readable instructions (e.g., representing (one or more) programs and / or (one or more) program elements, such as an operating system). Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by computing device 1400. As used herein, computer storage media does not include signals themselves.

[0217] Computer storage media can embody computer-readable instructions, data structures, program modules, and / or other data types in a modulated data signal such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" may refer to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, computer storage media can include wired media (such as a wired network or direct-wired connection) and wireless media (such as acoustic, RF, infrared, and other wireless media). Combinations of any of the above should also be included within the scope of computer-readable media.

[0218] The CPU 1406 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1400 to perform one or more of the methods and / or processes described herein. The CPUs 1406 may each include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of handling numerous software threads simultaneously. The CPU 1406 may include any type of processor and may include different types of processors depending on the type of computing device 1400 being implemented (e.g., a processor with fewer cores for a mobile device and a processor with more cores for a server). For example, depending on the type of computing device 1400, the processor may be an Advanced RISC Machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). The computing device 1400 may also include one or more CPUs 1406 in addition to one or more microprocessors or supplemental coprocessors (such as a math coprocessor).

[0219] In addition to or in lieu of CPU(s) 1406, GPU(s) 1408 may be configured to execute at least some of the computer-readable instructions to control one or more components of computing device 1400 to perform one or more of the methods and / or processes described herein. One or more of GPUs 1408 may be integrated GPUs (e.g., with one or more of CPUs 1406) and / or one or more of GPUs 1408 may be discrete GPUs. In embodiments, one or more of GPUs 1408 may be coprocessors for one or more of CPUs 1406. GPU 1408 may be used by computing device 1400 to render graphics (e.g., 3D graphics) or perform general-purpose computations. For example, GPU 1408 may be used for general-purpose computing on a GPU (GPGPU). GPU 1408 may include hundreds or thousands of cores capable of handling hundreds or thousands of software threads simultaneously. GPU 1408 may generate pixel data for an output image in response to rendering commands (e.g., rendering commands received from CPU 1406 via a host interface). GPU 1408 may include graphics memory (e.g., display memory) for storing pixel data or any other suitable data (e.g., GPGPU data). Display memory may be included as part of memory 1404. GPU 1408 may include two or more GPUs operating in parallel (e.g., via a link). The link may directly connect the GPUs (e.g., using NVLINK) or may connect the GPUs through a switch (e.g., using NVSwitch). When combined, each GPU 1408 may generate pixel data or GPGPU data for a different portion of an output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory or may share memory with other GPUs.

[0220] In addition to or in lieu of the CPU 1406 and / or GPU 1408, the logic unit 1420 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1400 to perform one or more of the methods and / or processes described herein. In embodiments, the CPU(s) 1406, the GPU(s) 1408, and / or the logic unit(s) 1420 may execute any combination of methods, processes, and / or portions thereof, either discretely or jointly. One or more of the logic units 1420 may be part of and / or integrated into one or more of the CPU 1406 and / or GPU 1408, and / or one or more of the logic units 1420 may be discrete components or otherwise external to the CPU 1406 and / or GPU 1408. In embodiments, one or more of logic units 1420 may be a co-processor to one or more of CPUs 1406 and / or one or more of GPUs 1408 .

[0221] Examples of logic unit 1420 include one or more processing cores and / or components thereof, such as a data processing unit (DPU), a tensor core (TC), a tensor processing unit (TPU), a pixel vision core (PVC), a vision processing unit (VPU), a graphics processing cluster (GPC), a texture processing cluster (TPC), a streaming multiprocessor (SM), a tree transverse unit (TTU), an artificial intelligence accelerator (AIA), a deep learning accelerator (DLA), an arithmetic logic unit (ALU), an application-specific integrated circuit (ASIC), a floating point unit (FPU), an input / output (I / O) element, a peripheral component interconnect (PCI) or a peripheral component interconnect express (PCIe) element, etc.

[0222] The communication interface 1410 may include one or more receivers, transmitters, and / or transceivers that allow the computing device 1400 to communicate with other computing devices via an electronic communication network (including wired and / or wireless communications). The communication interface 1410 may include components and functionality that allow communication over any of a number of different networks, such as a wireless network (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), a wired network (e.g., via Ethernet or Wi-Fi), a low-power wide-area network (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, the logic unit 1420 and / or the communication interface 1410 may include one or more data processing units (DPUs) to transmit data received over the network and / or through the interconnect system 1402 directly to one or more GPUs 1408 (e.g., memory of one or more GPUs 1408).

[0223] I / O ports 1412 can allow computing device 1400 to be logically coupled to other devices including I / O components 1414, (one or more) presentation components 1418, and / or other components, some of which can be built into (e.g., integrated into) computing device 1400. Illustrative I / O components 1414 include microphones, mice, keyboards, joysticks, game pads, game controllers, satellite dishes, scanners, printers, wireless devices, and the like. I / O components 1414 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological input generated by a user. In some cases, the input can be transmitted to an appropriate network element for further processing. The NUI can implement any combination of voice recognition, stylus recognition, facial recognition, biometric recognition, gesture recognition on and near the screen, air gestures, head and eye tracking, and touch recognition (as described in more detail below) associated with the display of computing device 1400. Computing device 1400 can include a depth camera for gesture detection and recognition, such as a stereo camera system, an infrared camera system, an RGB camera system, touch screen technology, and combinations thereof. Additionally, computing device 1400 may include an accelerometer or gyroscope that allows for detection of motion (e.g., as part of an inertial measurement unit (IMU)). In some examples, computing device 1400 may use the output of the accelerometer or gyroscope to render immersive augmented or virtual reality.

[0224] The power supply 1416 can include a hardwired power supply, a battery power supply, or a combination thereof. The power supply 1416 can provide power to the computing device 1400 to allow the components of the computing device 1400 to operate.

[0225] The presentation component 1418 may include a display (e.g., a monitor, a touch screen, a television screen, a head-up display (HUD), other display types, or a combination thereof), speakers, and / or other presentation components. The presentation component 1418 may receive data from other components (e.g., the GPU 1408, the CPU 1406, etc.) and output the data (e.g., as images, video, sound, etc.).

[0226] Sample Data Center

[0227] Figure 15 An example data center 1500 is shown that can be used in at least one embodiment of the present disclosure. The data center 1500 can include a data center infrastructure layer 1510, a framework layer 1520, a software layer 1530, and / or an application layer 1540.

[0228] like Figure 15 As shown, the data center infrastructure layer 1510 may include a resource coordinator 1512, grouped computing resources 1514, and node computing resources ("node CRs") 1516(1)-1516(N), where "N" represents any complete positive integer. In at least one embodiment, the node CRs 1516(1)-1516(N) may include, but is not limited to, any number of central processing units ("CPUs") or other processors (including accelerators, field programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memories), storage devices (e.g., solid-state or disk drives), network input / output ("NW I / O") devices, network switches, virtual machines ("VMs"), power modules and / or cooling modules, etc. In some embodiments, one or more of the node CRs 1516(1)-1516(N) may correspond to a server having one or more of the above-mentioned computing resources. Furthermore, in some embodiments, node CRs 1516(1)-15161(N) may include one or more virtual components, such as vGPUs, vCPUs, etc., and / or one or more of node CRs 1516(1)-1516(N) may correspond to a virtual machine (VM).

[0229] In at least one embodiment, the grouped computing resources 1514 may include separate groups of node CRs 1516 housed in one or more racks (not shown), or many racks housed in data centers at different geographical locations (also not shown). Separate groups of node CRs 1516 within the grouped computing resources 1514 may include grouped computing, network, memory, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, several node CRs 1516 including CPUs, GPUs, DPUs, and / or other processors may be grouped in one or more racks to provide computing resources to support one or more workloads. One or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0230] Resource coordinator 1512 may configure or otherwise control one or more node CRs 1516(1)-1516(N) and / or grouped computing resources 1514. In at least one embodiment, resource coordinator 1512 may comprise a software design infrastructure ("SDI") management entity for data center 1500. Resource coordinator 1512 may comprise hardware, software, or some combination thereof.

[0231] In at least one embodiment, Figure 15 As shown, the framework layer 1520 may include a job scheduler 1533, a configuration manager 1534, a resource manager 1536 and / or a distributed file system 1538. The framework layer 1520 may include a framework that supports software 1532 of the software layer 1530 and / or one or more applications 1542 of the application layer 1540. The software 1532 or the application 1542 may include network-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. The framework layer 1520 may be, but is not limited to, a free and open source software network application framework (such as Apache Spark) that can use the distributed file system 1538 for large-scale data processing (e.g., "big data"). TM(hereinafter referred to as "Spark"). In at least one embodiment, the job scheduler 1533 may include a Spark driver to facilitate scheduling workloads supported by the different layers of the data center 1500. The configuration manager 1534 may be capable of configuring the different layers, such as the software layer 1530 and the framework layer 1520 (which includes Spark and a distributed file system 1538 for supporting large-scale data processing). The resource manager 1536 may be capable of managing clustered or grouped computing resources that are mapped to the distributed file system 1538 and the job scheduler 1533 or allocated to support the distributed file system 1538 and the job scheduler 1533. In at least one embodiment, the clustered or grouped computing resources may include the grouped computing resources 1514 at the data center infrastructure layer 1510. The resource manager 1536 may coordinate with the resource coordinator 1512 to manage these mapped or allocated computing resources.

[0232] In at least one embodiment, the software 1532 included in the software layer 1530 may include software used by at least a portion of the node CRs 1516(1)-1516(N), the grouped computing resources 1514, and / or the distributed file system 1538 of the framework layer 1520. The one or more types of software may include, but are not limited to, Internet web search software, email virus scanning software, database software, and streaming video content software.

[0233] In at least one embodiment, the applications 1542 included in the application layer 1540 may include one or more types of applications used by at least a portion of the node CRs 1516(1)-1516(N), the grouped computing resources 1514, and / or the distributed file system 1538 of the framework layer 1520. The one or more types of applications may include, but are not limited to, any number of genomic applications, cognitive computing, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments.

[0234] In at least one embodiment, any of configuration manager 1534, resource manager 1536, and resource coordinator 1512 can implement any number and type of self-modification actions based on any amount and type of data acquired in any technically feasible manner. The self-modification actions can save a data center operator of data center 1500 from making potentially poor configuration decisions and potentially avoiding underutilized and / or poorly performing portions of the data center.

[0235] According to one or more embodiments described herein, data center 1500 may include tools, services, software, or other resources to train one or more machine learning models or use one or more machine learning models to predict or infer information. For example, the machine learning model(s) may be trained by computing weight parameters according to a neural network architecture using the software and / or computing resources described above with respect to data center 1500. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks may be used to infer or predict information using the resources described above with respect to data center 1500 using weight parameters computed using one or more training techniques, such as, but not limited to, those described herein.

[0236] In at least one embodiment, data center 1500 may use a CPU, an application-specific integrated circuit (ASIC), a GPU, an FPGA, and / or other hardware (or virtual computing resources corresponding thereto) to perform training and / or inference using the aforementioned resources. In addition, one or more of the software and / or hardware resources described above may be configured to allow a user to train or perform inference services on information, such as image recognition, speech recognition, or other artificial intelligence services.

[0237] Sample network environment

[0238] A network environment suitable for implementing embodiments of the present disclosure may include one or more client devices, servers, network attached storage (NAS), other backend devices, and / or other device types. The client devices, servers, and / or other device types (e.g., each device) may be configured to: Figure 14 The backend devices 1500 may be implemented on one or more instances of the computing device(s) 1400 - for example, each device may include similar components, features, and / or functionality of the computing device(s) 1400. In addition, in the case of implementing backend devices (e.g., servers, NAS, etc.), the backend devices may be included as part of the data center 1500, examples of which are discussed herein with respect to FIG. Figure 15 Describe in more detail.

[0239] The components of the network environment can communicate with each other via a network, which can be wired, wireless, or both. The network can include multiple networks or one of multiple networks. For example, the network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. In the case where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) can provide wireless connections.

[0240] Compatible network environments may include one or more peer-to-peer network environments (in which case the server may not be included in the network environment) and one or more client-server network environments (in which case one or more servers may be included in the network environment). In a peer-to-peer network environment, the functionality described herein for the server may be implemented on any number of client devices.

[0241] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, and the like. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework for software supporting the software layer and / or one or more applications in the application layer. The software or application may include network-based service software or applications, respectively. In an embodiment, one or more client devices may use network-based service software or applications (e.g., by accessing the service software and / or application via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a free and open source software network application framework that can use a distributed file system for large-scale data processing (e.g., "big data").

[0242] A cloud-based network environment can provide cloud computing and / or cloud storage that performs any combination of the computing and / or data storage functions described herein (or one or more portions thereof). Any of these different functions can be distributed across multiple locations from a central or core server (e.g., one or more data centers that can be distributed across a state, region, country, global, etc.). If the connection to the user (e.g., client device) is relatively close to an edge server, the core server can assign at least a portion of the functionality to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0243] The client device(s) may include the Figure 14At least some of the components, features, and functionality of the described example computing device(s) 1400. By way of example and not limitation, the client device may be implemented as a personal computer (PC), a laptop computer, a mobile device, a smartphone, a tablet computer, a smartwatch, a wearable computer, a personal digital assistant (PDA), an MP3 player, a virtual reality headset, a global positioning system (GPS) or device, a video player, a camera, a surveillance device or system, a vehicle, a boat, a spacecraft, a virtual machine, a drone, a robot, a handheld communication device, a hospital device, a gaming device or system, an entertainment system, a vehicle computer system, an embedded system controller, a remote control, an appliance, a consumer electronic device, a workstation, an edge device, any combination of the depicted devices, or any other suitable device.

[0244] The present disclosure can be described in the general context of machine-usable instructions or computer code executed by a computer or other machine such as a personal digital assistant or other handheld device, including computer-executable instructions such as program modules. Generally, program modules including routines, programs, objects, components, data structures, etc. refer to code that performs a specific task or implements a specific abstract data type. The present disclosure can be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc. The present disclosure can also be practiced in a distributed computing environment where tasks are performed by remote processing devices linked through a communication network.

[0245] As used herein, the phrase "and / or" with respect to two or more elements should be interpreted as referring to only one element or combination of elements. For example, "element A, element B, and / or element C" may include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. In addition, "at least one of element A or element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Further, "at least one of element A and element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0246] The subject matter of the present disclosure is described in detail herein to meet statutory requirements. However, the description itself is not intended to limit the scope of the present disclosure. On the contrary, the present inventors have contemplated that the claimed subject matter may also be embodied in other ways to include steps that are different from the steps described herein in conjunction with other current or future technologies, or combinations of similar steps. Moreover, although the terms "step" and / or "block" may be used herein to imply different elements of the method employed, these terms should not be interpreted as implying any particular order among or between the various steps disclosed herein, unless the order of the steps is explicitly described.

[0247] Sample text support

[0248] The disclosure of this application also includes the following numbered clauses:

[0249] Item 1. One or more processors comprising processing circuitry for detecting one or more hazards in an environment based at least on one or more neural networks (NNs) comprising one or more converters processing representations of image data and LiDAR data corresponding to an environment of the self-machine.

[0250] Clause 2. The one or more processors of Clause 1, wherein the processing circuitry is further configured to control one or more operations of the self-machine based at least on the one or more hazards.

[0251] Clause 3. One or more processors as described in clause 1 or 2, wherein processing the representation of the image data includes: generating one or more three-dimensional converter queries based at least on one or more two-dimensional candidate enclosing shapes extracted from the image data using the one or more NNs, and applying one or more sampled two-dimensional image features extracted from the image data using the one or more NNs to the one or more converters.

[0252] Clause 4. One or more processors as described in clause 1 or 2, wherein processing the representation of the LiDAR data includes: generating one or more three-dimensional converter queries based at least on one or more two-dimensional candidate bounding shapes extracted from the LiDAR data using the one or more NNs, and applying one or more sampled two-dimensional LiDAR features extracted from the LiDAR data using the one or more NNs to the one or more converters.

[0253] Clause 5. One or more processors as described in clause 1 or 2, wherein processing the representations of the image data and the LiDAR data includes: using one or more cross-attention layers of the one or more converters to fuse one or more sampled two-dimensional image features and one or more sampled two-dimensional LiDAR features.

[0254] Clause 6. One or more processors as described in clause 1 or 2, wherein processing the representations of the image data and the LiDAR data includes: projecting one or more key points associated with one or more reference three-dimensional positions corresponding to one or more converter queries into the extracted image features and the extracted LiDAR features.

[0255] Clause 7. One or more processors as described in clause 1 or 2, wherein the processing circuit is further used to: generate multiple converter queries based on at least one or more candidate bounding shapes extracted from the image data or the LiDAR data, one or more randomly initialized three-dimensional positions, and one or more self-motion compensated converter predictions.

[0256] Clause 8. One or more processors as described in clause 1 or 2, wherein the processing circuit is further used to: detect the one or more hazards based at least on the one or more converters by performing the following operations: generating classification data indicating whether there is predicted road debris at a three-dimensional location corresponding to each converter query in the one or more converter queries, and regressing a representation of the enclosing shape of the road debris at the three-dimensional location.

[0257] Clause 9. The one or more processors of clause 1 or 2, wherein the processing circuitry is further configured to update the one or more transformers based at least on omitting transformer queries representing reference three-dimensional positions outside of the ground truth navigable space.

[0258] Clause 10. The one or more processors of Clause 1 or 2, wherein the processing circuitry is further configured to: navigate the self-machine based at least on avoiding or compensating for the one or more hazards.

[0259] Clause 11. The one or more processors of clause 1 or 2, wherein the one or more processors are included in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing optical transport simulation; a system for performing collaborative content creation of 3D assets; a system for performing deep learning operations; a system for performing teleoperations; a system for performing real-time streaming; a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; a system implemented using edge devices; a system implemented using robots; a system for performing conversational AI operations; a system implementing one or more language models; a system implementing one or more large language models (LLMs); a system implementing one or more visual language models (VLMs); a system implementing one or more multi-modal language models; a system for generating synthetic data; a system for generating synthetic data using AI; a system containing one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

[0260] Clause 12. A system comprising one or more processors to: control one or more operations of a self machine in an environment based at least on one or more hazards detected based at least on processing, by one or more neural networks (NNs) including one or more transformers, a representation of image data and LiDAR data corresponding to the environment.

[0261] Clause 13. A method comprising: detecting one or more static scene points represented by one or more LiDAR detections.

[0262] Clause 14. The method of clause 13, further comprising: detecting one or more static hazards represented by the one or more static scene points.

[0263] Clause 15. The method of clause 13 or 14, further comprising: generating a ground truth representation of the one or more static hazards.

[0264] Clause 16. The method of clause 13, 14, or 15, wherein detecting the one or more static scene points is based at least on a consistency measure of at least one of a presence or a distance of the one or more LiDAR detections in a set of projected images.

[0265] Clause 17. The method of clause 13, 14, or 15, wherein detecting the one or more static scene points comprises filtering out one or more non-static points from the one or more LiDAR detections.

[0266] Clause 18. The method of clause 13, 14 or 15 further comprising detecting the one or more static hazards based at least on evaluating the ground truth detected from the one or more LiDAR detections and the one or more static scene points using an occupancy scoring function based on height and curvature.

[0267] Clause 19. The method of clause 13, 14 or 15 further comprises detecting the one or more static hazards based at least on a segmented occupancy grid generated based at least on the ground truth detected from the one or more LiDAR detections and the one or more static scene points.

[0268] Clause 20. The method of clause 13, 14, or 15, further comprising detecting the one or more static hazards based at least on extracting one or more contours from the segmented occupancy map.

[0269] Clause 21. The method of clause 13, 14, or 15, further comprising detecting the one or more static hazards based at least on extracting one or more child contours of a parent contour representing the predicted ground-truth navigable space from the segmented occupancy map.

[0270] Clause 22. A method as described in clause 13, 14 or 15, wherein generating the true value representation of the one or more static hazards is based at least on associating a range of height values ​​corresponding to a set of the LiDAR detections enclosed by each of the one or more extracted two-dimensional contours representing the one or more static hazards.

[0271] Clause 23. The method of clause 13, 14, or 15, further comprising detecting a representation of the ground truth navigable space based at least on evaluating the ground truth ground detected from the one or more LiDAR detections and the one or more static scene points.

[0272] Clause 24. The method of clause 13, 14, or 15, further comprising updating one or more static hazard detection networks based at least on the ground-truth representation of the one or more static hazards.

[0273] Clause 25. The method of clause 13, 14, or 15, wherein the method is performed by at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing optical transport simulation; a system for performing collaborative content creation of 3D assets; a system for performing deep learning operations; a system for performing teleoperations; a system for performing real-time streaming; a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; a system implemented using edge devices; a system implemented using robots; a system for performing conversational AI operations; a system implementing one or more language models; a system implementing one or more large language models (LLMs); a system implementing one or more visual language models (VLMs); a system implementing one or more multi-modal language models; a system for generating synthetic data; a system for generating synthetic data using AI; a system including one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

[0274] Clause 26. One or more processors comprising processing circuitry to detect one or more features of a surface in an environment corresponding to self-machine based at least on processing representations of image data and LiDAR data corresponding to the environment comprising one or more transformers of one or more neural networks (NNs).

[0275] Clause 27. The one or more processors of clause 26, wherein the processing circuitry is further to control one or more operations of the self-machine based at least on the one or more features of the surface.

[0276] Clause 28. The one or more processors of clause 26 or 27, wherein the circuitry is further to generate a plurality of three-dimensional transformer queries based at least on one or more sampling points on the surface and one or more self-motion compensated transformer predictions.

[0277] Clause 29. The one or more processors of clause 26 or 27, wherein the circuitry is further to generate one or more three-dimensional transformer queries representing one or more three-dimensional locations based at least on one or more trajectories of the self-machine.

[0278] Clause 30. The one or more processors of clause 26 or 27, wherein the circuitry is further to generate one or more three-dimensional transformer queries representing one or more three-dimensional locations based at least on logarithmically sampling one or more trajectories of the self-machine.

[0279] Clause 31. The one or more processors of clause 26 or 27, wherein processing the image data and the representation of the LiDAR data comprises refining one or more initial heights of the surface represented by one or more initial three-dimensional transformer queries based at least on fusing one or more sampled two-dimensional image features and one or more sampled two-dimensional LiDAR features in one or more cross-attention layers of the one or more transformers.

[0280] Clause 32. The one or more processors of clause 26 or 27, wherein the circuitry is further to project one or more key points associated with one or more reference three-dimensional locations into the extracted image features and the extracted LiDAR features, the one or more reference three-dimensional locations corresponding to one or more transformer queries representing one or more initial heights of the surface.

[0281] Clause 33. The one or more processors of clause 26 or 27, wherein the circuitry is further to detect the one or more features of the surface based at least on the one or more transformers performing the following operations: regressing a representation of one or more height values of one or more sampled points of the surface corresponding to each of one or more transformer queries.

[0282] Clause 34. The one or more processors of clause 26 or 27, wherein the one or more NNs form a multi-task network comprising a first transformer output head that regresses one or more surface curves of the surface and a second transformer output head that regresses one or more enclosing shapes of detected road debris on the surface.

[0283] Clause 35. The one or more processors of clause 26 or 27, wherein the one or more operations comprise at least one of the following: avoiding one or more detected bumps represented by one or more features of the surface, adjusting a suspension of the ego machine based at least on surface curves represented by one or more features of the surface, or applying an anticipatory acceleration or deceleration based at least on an approaching surface slope represented by one or more features of the surface.

[0284] Clause 36. One or more processors as described in clause 26 or 27, wherein the one or more processors are included in at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulations; a system for performing collaborative content creation of 3D assets; a system for performing deep learning operations; a system for performing remote operations; a system for performing real-time streaming; a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; a system implemented using an edge device; a system implemented using a robot; a system for performing conversational AI operations; a system implementing one or more language models; a system implementing one or more large language models (LLMs); a system implementing one or more visual language models (VLMs); a system implementing one or more multimodal language models; a system for generating synthetic data; a system for generating synthetic data using AI; a system comprising one or more virtual machines (VMs); a system implemented at least in part in a data center; or a system implemented at least in part using cloud computing resources.

[0285] Clause 37. A system comprising one or more processors configured to control one or more operations of a self-machine based at least on one or more features of a surface in an environment, the one or more features detected based at least on one or more neural networks (NNs) comprising one or more converters processing representations of image data and LiDAR data corresponding to the environment.

[0286] Clause 38. A system as described in Clause 37, wherein the system is included in at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulations; a system for performing collaborative content creation of 3D assets; a system for performing deep learning operations; a system for performing remote operations; a system for performing real-time streaming; a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; a system implemented using an edge device; a system implemented using a robot; a system for performing conversational AI operations; a system implementing one or more language models; a system implementing one or more large language models (LLMs); a system implementing one or more visual language models (VLMs); a system implementing one or more multimodal language models; a system for generating synthetic data; a system for generating synthetic data using AI; a system comprising one or more virtual machines (VMs); a system implemented at least in part in a data center; or a system implemented at least in part using cloud computing resources.

[0287] Clause 39. A method comprising: generating a representation of a detected ground surface based at least on one or more LiDAR detections of an ego machine.

[0288] Clause 40. The method of Clause 39, further comprising generating a representation based at least on one or more sampling points of one or more trajectory samples of the ego machine.

[0289] Clause 41. The method of clause 39 or 40, further comprising: generating a ground truth representation of one or more features of the detected ground surface at the one or more sampling points.

[0290] Clause 42. The method of clause 39, 40, or 41, further comprising: accumulating the one or more LiDAR detections of the detected ground using one or more LiDAR sensors of one or more data collection vehicles and one or more fixed LiDAR sensors.

[0291] Clause 43. The method of clause 39, 40, or 41, further comprising: applying a smoothing process to the area of ​​the detected ground before sampling the one or more sampling points from the area of ​​the detected ground including the one or more trajectories of the ego machine.

[0292] Clause 44. The method of clause 39, 40, or 41, wherein the one or more characteristics of the detected ground surface include one or more detected heights of the detected ground surface.

[0293] Clause 45. The method as described in clause 39, 40 or 41 further includes: associating one or more tags representing one or more detected true weather conditions with one or more frames of the one or more LiDAR detections based at least on a power distribution of a set of non-static scene points detected from the one or more LiDAR detections in a specified volume.

[0294] Clause 46. The method as described in clause 39, 40 or 41 further includes: associating one or more labels representing one or more detected true surface conditions with one or more frames of the one or more LiDAR detections based at least on the power distribution of a set of non-static scene points detected from the one or more LiDAR detections in a specified area of ​​the detected ground.

[0295] Clause 47. The method of clause 39, 40, or 41, further comprising generating one or more surface curve detection networks based at least on the ground truth representation of the one or more features of the detected ground surface at the one or more sampling points.

[0296] Clause 48. A method as described in clauses 39, 40 or 41, wherein the method is performed by at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulations; a system for performing collaborative content creation of 3D assets; a system for performing deep learning operations; a system for performing remote operations; a system for performing real-time streaming; a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; a system implemented using an edge device; a system implemented using a robot; a system for performing conversational AI operations; a system implementing one or more language models; a system implementing one or more large language models (LLMs); a system implementing one or more visual language models (VLMs); a system implementing one or more multimodal language models; a system for generating synthetic data; a system for generating synthetic data using AI; a system comprising one or more virtual machines (VMs); a system implemented at least in part in a data center; or a system implemented at least in part using cloud computing resources.

Claims

1. One or more processors, including processing circuitry configured to: detecting one or more hazards in the environment based at least on one or more neural networks (NN) including one or more transducers processing image data and a representation of LiDAR data corresponding to the environment of the ego machine; and One or more operations of the self-machine are controlled based at least on the one or more hazards.

2. One or more processors according to claim 1, wherein: Processing the representation of the image data includes generating one or more three-dimensional transformer queries based at least on one or more two-dimensional candidate bounding shapes extracted from the image data using the one or more NNs, and applying one or more sampled two-dimensional image features extracted from the image data using the one or more NNs to the one or more transformers.

3. The one or more processors of claim 1 , wherein: Processing the representation of the LiDAR data includes generating one or more three-dimensional transformer queries based at least on one or more two-dimensional candidate bounding shapes extracted from the LiDAR data using the one or more NNs, and applying one or more sampled two-dimensional LiDAR features extracted from the LiDAR data using the one or more NNs to the one or more transformers.

4. The one or more processors of claim 1 , wherein: Processing the representations of the image data and the LiDAR data includes fusing one or more sampled two-dimensional image features and one or more sampled two-dimensional LiDAR features using one or more crisscross attention layers of the one or more transformers.

5. The one or more processors of claim 1 , wherein: Processing the representations of the image data and the LiDAR data includes projecting one or more keypoints associated with one or more reference three-dimensional locations corresponding to one or more converter queries into the extracted image features and the extracted LiDAR features.

6. The one or more processors of claim 1 , wherein: The processing circuit is further configured to generate a plurality of converter queries based on at least one or more candidate bounding shapes extracted from the image data or the LiDAR data, one or more randomly initialized three-dimensional positions, and one or more ego-motion compensated converter predictions.

7. The one or more processors of claim 1 , wherein: The processing circuitry is further configured to detect the one or more hazards based at least on the one or more transducers by generating classification data representing the presence or absence of predicted road debris at a three-dimensional location corresponding to each of the one or more transducer queries, and regressing a representation of an enclosing shape of the road debris at the three-dimensional location.

8. The one or more processors of claim 1 , wherein: The processing circuitry is further configured to update the one or more transducers based at least on omitting transducer queries representing reference three-dimensional positions outside of the ground-truth navigable space.

9. The one or more processors of claim 1 , wherein: The processing circuit is further configured to navigate the self-machine based at least on avoiding or compensating for the one or more hazards.

10. The one or more processors of claim 1, wherein: The one or more processors are included in at least one of the following: control systems for autonomous or semi-autonomous machines; Perception systems for autonomous or semi-autonomous machines; a system for performing simulation operations; Systems for performing digital twin operations; a system for performing light transport simulations; A system for performing collaborative content creation of 3D assets; Systems for performing deep learning operations; systems for performing remote operations; A system for performing real-time streaming; Systems for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; Systems implemented using edge devices; Systems implemented using robots; Systems for performing conversational AI operations; a system implementing one or more language models; A system implementing one or more Large Language Models (LLMs); A system implementing one or more visual language models (VLMs); a system implementing one or more multimodal language models; Systems for generating synthetic data; Systems for generating synthetic data using AI; A system comprising one or more virtual machines VM; A system implemented at least in part in a data center; or A system implemented at least in part using cloud computing resources.

11. A system comprising one or more processors configured to control one or more operations of a self-machine in an environment based at least on one or more hazards detected based at least on one or more neural networks (NN) comprising one or more transducers processing representations of image data and LiDAR data corresponding to the environment.

12. A method comprising: detecting one or more static scene points represented by one or more LiDAR detections; detecting one or more static hazards represented by the one or more static scene points; as well as A ground truth representation of the one or more static hazards is generated.

13. The method according to claim 12, wherein: Detecting the one or more static scene points is based at least on a consistency measure of at least one of presence or distance of the one or more LiDAR detections in a set of projected images.

14. The method according to claim 12, wherein: Detecting the one or more static scene points includes filtering out one or more non-static points from the one or more LiDAR detections.

15. The method according to claim 12, further comprising: The one or more static hazards are detected based at least on evaluating the ground truth detected from the one or more LiDAR detections and the one or more static scene points using an altitude and curvature based occupancy scoring function.

16. The method according to claim 12, further comprising: The one or more static hazards are detected based at least on a segmented occupancy grid generated based at least on a ground truth detected from the one or more LiDAR detections and the one or more static scene points.

17. The method according to claim 12, further comprising: The one or more static hazards are detected based at least on extracting one or more contours from the segmented occupancy map.

18. The method according to claim 12, further comprising: The one or more static hazards are detected based at least on extracting one or more child contours of a parent contour representing the predicted ground-truth navigable space from the segmented occupancy map.

19. The method according to claim 12, wherein: Generating the ground-truth representation of the one or more static hazards is based at least on associating a range of height values ​​corresponding to a set of the LiDAR detections encompassed by each of one or more extracted two-dimensional contours representing the one or more static hazards.

20. The method of claim 12, further comprising: A representation of the ground-truth navigable space is detected based at least on evaluating the ground-truth ground detected from the one or more LiDAR detections and the one or more static scene points.

21. The method according to claim 12, further comprising: One or more static hazard detection networks are updated based at least on the ground-truth representations of the one or more static hazards.

22. The method according to claim 12, wherein The method is performed by at least one of the following: control systems for autonomous or semi-autonomous machines; Perception systems for autonomous or semi-autonomous machines; a system for performing simulation operations; Systems for performing digital twin operations; a system for performing light transport simulations; A system for performing collaborative content creation of 3D assets; Systems for performing deep learning operations; systems for performing remote operations; A system for performing real-time streaming; Systems for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; Systems implemented using edge devices; Systems implemented using robots; Systems for performing conversational AI operations; a system implementing one or more language models; A system implementing one or more Large Language Models (LLMs); A system implementing one or more visual language models (VLMs); a system implementing one or more multimodal language models; Systems for generating synthetic data; Systems for generating synthetic data using AI; A system comprising one or more virtual machines VM; A system implemented at least in part in a data center; or A system implemented at least in part using cloud computing resources.

Images