Multi-view LiDAR perception with motion cues for autonomous machines and applications

Abstract

Embodiments of the present disclosure relate to multi-view LiDAR perception with motion cues for autonomous machines and applications, in particular to multi-view LiDAR perception with motion cues for autonomous and semi-autonomous machines and applications. The DNN may be used to detect objects, airworthiness space, weather or surface conditions, artifacts, and / or other portions or features of the environment based on multiple views of LiDAR data from multiple time slices. The DNN may include a plurality of input channels for processing a plurality of views of sensor data from a plurality of time slices to provide motion cues; the extracted features from different time slices may be geometrically projected from the first 2D view to the second 2D view and combined with features extracted from the second 2D view, and then applied to subsequent stages of the DNN. The data generated by the DNN may be provided to the driving stack of an autonomous vehicle or other self-machine to enable safe planning and control of the vehicle.

Description

Background Technology

[0001] Designing a system capable of safely driving a vehicle autonomously without supervision is extremely difficult. An autonomous vehicle should at least be functionally equivalent to an attentive driver (relying on a perception and action system with a strong ability to identify and react to moving and static obstacles in complex environments) to avoid collisions with other objects or structures in its path. Therefore, the ability to detect instances of moving objects (e.g., cars, pedestrians, etc.) and environmental components is generally crucial for autonomous driving perception systems. This capability becomes increasingly important as the operating environments of autonomous vehicles expand from highway environments to semi-urban and urban environments characterized by complex scenes with numerous obstructions and intricate shapes.

[0002] Conventional perception methods typically rely on cameras or LiDAR (Light Detection and Ranging) sensors to detect objects and parts of the environment, and various methods using deep neural networks (DNNs) have been developed to perform LiDAR and camera perception. However, the state-of-the-art detection accuracy still has limitations. For example, conventional DNNs have limited resolution, which can lead to limited accuracy in detecting objects or objects, resulting in inappropriate navigation decisions that increase the risk of collisions or off-road accidents. Furthermore, conventional DNNs may miss minor hazards such as debris, potholes, or uneven road surfaces, potentially leading to vehicle damage, accidents, or uncomfortable driving conditions. Limited accuracy in detecting weather or road conditions can lead to inappropriate driving responses, such as insufficient braking on icy roads or inability to adjust speed during heavy rain, potentially jeopardizing vehicle and passenger safety. In addition, sensor artifacts and environmental artifacts (e.g., weather particles such as dust, snow, and rain) can cause erroneous readings or obscure real objects, leading to unnecessary or incorrect navigation or evasive maneuvers, or failure to detect actual hazards, potentially compromising safety. Therefore, there is a need to improve perception technologies for autonomous driving applications. Summary of the Invention

[0003] Embodiments of this disclosure relate to multi-view LiDAR perception utilizing motion cues for autonomous and semi-autonomous machines and applications. Compared to conventional systems, DNNs can be used to detect objects, navigable space, weather or surface conditions, artifacts, and / or other parts or features of the environment based on multiple views of LiDAR data from multiple time slices.

[0004] More specifically, a DNN may include: a first stage that processes multiple two-dimensional (2D) views of a 3D environment; a second stage that processes features extracted from a common (e.g., top-down) 2D view; and any number of output heads (e.g., derived from the first or second stage). The DNN may include multiple input channels for processing sensor data from multiple time slices to provide motion cues; and features extracted from different time slices may be geometrically projected from the first 2D view onto a second 2D view, combined with features extracted from the second 2D view, and then applied to subsequent stages of the DNN. Example output heads include: an artifact detection head that processes features extracted from a perspective view (e.g., a non-destructive distance image); and an object detection head, an airworthiness space detection head, and / or a weather and surface condition detection head that processes features extracted from multiple views and / or multiple time slices. In some embodiments, the DNN may process sensor data (e.g., LiDAR scans) sequentially at a specified frame rate, and the extracted features may be cached and used in multiple inferences.

[0005] Unlike some conventional geometric projection techniques that bin 3D points projected onto the same pixel or grid cell, in some embodiments, the characteristics of multiple returns are stored in a corresponding layer for each pixel or grid cell. Therefore, the resulting projected image can be considered to provide a (e.g., substantially) lossless way to align sensor data with a 2D grid and support multiple returns (and thus can be viewed as a lossless or multi-return image), and the characteristics of each return can be preserved and fed to a DNN, which can learn to identify and ignore zero-fill channels, focus on meaningful returns, and / or otherwise benefit from the increased input resolution, thereby improving accuracy compared to existing techniques.

[0006] Data generated by the output head, or its representation (e.g., detected objects, airworthiness space, weather and / or road conditions, and / or artifacts), can be provided to the driving stack of an autonomous or semi-autonomous vehicle or other self-driving machine to enable safe planning and control of the autonomous or semi-autonomous vehicle. For example, the driving stack can use detected obstacles and detected airworthiness space to guide path planning and maneuvering decisions, use detected weather and / or road conditions to guide speed and handling strategies (e.g., slowing down on wet or icy roads), or monitor its Operational Design Domain (ODD) (e.g., ensuring the vehicle operates under predefined conditions designed and validated for it, adjusting operational behavior such as slowing down in heavy rain or avoiding certain maneuvers on icy or wet roads, alerting the driver to take over the vehicle when conditions exceed the vehicle's safe operating parameters), and / or can use detected artifacts to filter out points that do not belong to real surfaces or objects to prevent false positives (e.g., false braking due to incorrectly detected obstacles, false lane departure warnings due to misinterpreted road markings, false collision warnings due to incorrectly identified objects (such as overhanging branches or roadside debris).

[0007] Therefore, the techniques described herein can be used to detect and classify objects and / or parts or features of activities in an environment, and these detections and classifications can be provided to autonomous and / or semi-autonomous vehicle or machine driving stacks to enable safety planning and control of vehicles or machines. Attached Figure Description

[0008] The following describes in detail, with reference to the accompanying drawings, the system and method of multi-view LiDAR perception utilizing motion cues for autonomous and semi-autonomous machines and applications, wherein:

[0009] Figure 1 This is a data flow diagram illustrating an example detection pipeline according to some embodiments of the present disclosure;

[0010] Figure 2A This is an illustration of an example neural network that can be used to process multiple views of sensor data from multiple time slices, according to some embodiments of this disclosure. Figure 2B These are illustrations of example lossless projection images according to some embodiments of the present disclosure;

[0011] Figure 3 This is a diagram illustrating an example architecture of a neural network according to some embodiments of the present disclosure;

[0012] Figure 4 This is an illustration of an example neural network that can cache extracted features according to some embodiments of the present disclosure;

[0013] Figure 5This is a flowchart illustrating a method for multi-view LiDAR perception according to some embodiments of the present disclosure;

[0014] Figure 6A These are illustrations of exemplary autonomous vehicles according to some embodiments of the present disclosure;

[0015] Figure 6B According to some embodiments of this disclosure Figure 6A An example of camera position and field of view for an exemplary autonomous vehicle;

[0016] Figure 6C According to some embodiments of this disclosure Figure 6A A block diagram of an exemplary system architecture for an exemplary autonomous vehicle;

[0017] Figure 6D It is one or more cloud-based servers according to some embodiments of this disclosure and Figure 6A An exemplary system diagram of communication between autonomous vehicles;

[0018] Figure 7 This is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure; and

[0019] Figure 8 This is a block diagram of an example data center applicable to implementing some embodiments of this disclosure. Detailed Implementation

[0020] Systems and methods related to multi-view LIDAR perception utilizing motion cues for autonomous and semi-autonomous machines and applications are disclosed. This technology can be used to detect objects, airworthy spaces, weather or surface conditions, artifacts, and / or other parts or features of the environment, and can be used by autonomous vehicles, semi-autonomous vehicles, robots, and / or other types of objects or machines.

[0021] Although this disclosure may relate to example autonomous or semi-autonomous vehicles or machines 600 (also referred to herein as "vehicle 600" or "self-machine 600"), examples of which relate to Figures 6A-6DThe description herein is provided, but is not intended to be limiting. For example, the systems and methods described herein can be used with (but are not limited to) non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more advanced driver assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, aircraft, ships, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, engineering vehicles, trains, underwater vehicles, remotely controlled vehicles (e.g., drones), and / or other types of vehicles. Furthermore, although this disclosure describes perception with respect to autonomous driving, it is not intended to be limiting, and the systems and methods described herein can be used in augmented reality, virtual reality, mixed reality, robotics, security and supervision, autonomous or semi-autonomous machine applications, and / or any other technological space where perception can be used.

[0022] In some embodiments, a DNN can be used to process multiple views of LiDAR data and / or other sensor data representing a three-dimensional (3D) environment. The DNN may include: a first stage that processes multiple two-dimensional (2D) views of the 3D environment; a second stage that processes extracted features from a common (e.g., top-down) 2D view; and any number of output heads (e.g., derived from the first or second stage). The DNN may include multiple input channels for processing sensor data from multiple time slices to provide motion cues; and extracted features from different time slices may be geometrically projected from the first 2D view onto a second 2D view, combined with features extracted from the second 2D view, and applied to subsequent stages of the DNN. Example output heads include: an artifact detection head that processes features extracted from a perspective view (e.g., a non-destructive distance image); and an object detection head, an airworthiness space detection head, and / or a weather and surface condition detection head that processes features extracted from multiple views and / or multiple time slices.

[0023] In some embodiments, the input to the DNN can be formed by geometrically projecting LiDAR detection results into one or more projected images (e.g., non-destructive LiDAR distance images, top-view or bird's-eye view (BEV) images) and / or data from other sensors (e.g., images from any number of cameras). Sensor data representing multiple views (e.g., top-down views and non-destructive views) and / or multiple time slices (e.g., non-destructive LiDAR distance images representing multiple time slices) can be combined (e.g., stacked) and applied to the corresponding input channels of the DNN, and a first stage of the DNN can extract corresponding features (e.g., learned intermediate representations). The extracted features (e.g., distance images) for multiple time slices can be geometrically projected from a first 2D (e.g., perspective) view to a second 2D view, self-motion compensated, combined with each other (e.g., summed, averaged, cascaded, fused by a neural network, etc.), and combined with extracted features (e.g., bird's-eye view) extracted from the second 2D (e.g., bird's-eye view) view, and applied to subsequent stages of the DNN.

[0024] In some embodiments, the DNN can process sensor data (e.g., LiDAR scans) sequentially at a specified frame rate, and the extracted features can be cached and used in multiple inferences. Taking an example embodiment with a number of input channels for accepting projected images representing corresponding time slices (e.g., three input channels for three consecutive lossless LiDAR distance images), features can be extracted from the projected image of the current time slice and cached. The extracted features cached from one or more previous time slices can be retrieved, and the extracted features cached from one or more previous time slices can be aligned with the extracted features of the current time slice by performing self-motion compensation on the extracted features of previous time slices to reflect the current position and orientation of the sensor (e.g., transforming previously detected distance values ​​to compensate for the updated position and orientation of the moving sensor); and the extracted features of the current time slice and previous time slices can be geometrically projected onto a common coordinate system. This avoids repetitive feature extraction operations in continuous inference, thereby reducing computational requirements and accelerating processing time.

[0025] In some embodiments, one or more projected images processed using a DNN can encode a lossless view of the corresponding sensor data. Typically, LiDAR sensors can emit laser pulses that reflect off objects and surfaces and return to the sensor. The time taken for each pulse to return can be used to calculate distances, and the known angle and distance at which each pulse was emitted can be used to calculate the coordinates of the corresponding detected point. If a single laser pulse encounters multiple surfaces at different distances, the single laser pulse can produce multiple echoes, and the number and characteristics of the echoes can depend on the area of ​​the diverging beam and the transmittance of the surfaces it interacts with (e.g., the pulse may reflect from weather artifacts such as fog, producing a first echo, and may pass through the fog and reflect from the real surface behind the fog to produce a second echo; part of the beam may reflect from the first surface, while another part of the beam may reflect from the second surface; and so on). Therefore, some LiDAR sensors support multiple echoes (e.g., the same pulse can generate multiple 3D points).

[0026] Unlike some conventional geometric projection techniques that bin 3D points projected onto the same pixel or grid cell, in some embodiments, each pixel or grid cell stores characteristics of multiple echoes in a corresponding layer. For example, each pixel or grid cell may store a list or array of all 3D points projected onto it, including attributes for each point such as distance (range), echo signal strength, number of echoes, etc. In some embodiments, the projected image comprises multiple layers corresponding to the number of attributes for each echo supported by the LiDAR sensor (e.g., three attributes for each of three echoes), the maximum expected number of points that can be projected onto the pixel, and / or other parameters. Zero-padding can be used for pixels with fewer projected points or echoes than the supported number of points or echoes.

[0027] Therefore, the resulting projected image can be considered as providing a (e.g., essentially) lossless way to align sensor data with a 2D grid and support multiple echoes (and thus can be viewed as a lossless or multi-echo image). This type of data structure can be particularly useful when representing sensor data generated using foveated sensors, where the beams have non-linear spacing and the resolution at the center of the sensor's field of view is increased (e.g., doubled) (e.g., higher-resolution regions with higher point density in the point cloud are not lost or compressed when projected onto a 2D grid, thus generating projected images supporting different resolutions in different regions). Therefore, the characteristics of each echo can be preserved and fed to the DNN, which can learn to identify and ignore zero-fill channels, focus on meaningful echoes, and / or otherwise benefit from the increased input resolution, thereby improving accuracy compared to existing techniques.

[0028] In some embodiments, intermediate representations of features extracted from a first 2D (e.g., perspective) view can be geometrically projected and combined with extracted (e.g., bird's-eye view) features from a second 2D (e.g., bird's-eye view) view, and then applied to subsequent stages of the DNN. Projecting the learned intermediate representations improves performance compared to existing techniques that project per-pixel classification from a first 2D view to a second 2D view, eliminates the need for explicit category supervision at this stage, and eliminates the need for ground truth labels (e.g., per-pixel classification) at this stage.

[0029] Turning now to some example output heads, in some embodiments, features extracted in a common (e.g., top-down) 2D view (e.g., features extracted in the common 2D view stacked with or otherwise combined with extracted features projected into the common 2D view) may be applied to a second stage of the DNN (e.g., including encoder-decoder), and the output of the second stage may be applied to one or more output heads, such as object detection heads and / or airworthiness space detection heads, weather and surface condition detection heads, etc.

[0030] Object detection heads can extract classification data (e.g., confidence maps for any number of classes) and / or regress various types of information about detected objects (e.g., location, shape, orientation), which can be decoded to generate 2D or 3D bounding boxes (or other bounding shapes) and class labels for the detected objects. Airworthiness space detection heads can predict unobstructed areas in roads or other airworthiness spaces (e.g., represented by a top-down occupancy grid). In example applications for autonomous vehicles, DNNs can be used to predict one or more bounding boxes (e.g., 2D bounding boxes, 3D bounding boxes in a top-down view) for each detected object on a road or sidewalk, the class label for each detected object, and a 2D mask dividing drivable space, sidewalks, buildings, trees, poles, and / or other static environmental portions (e.g., in a top-down view).

[0031] The weather and surface condition detection head can classify one or more weather and / or surface conditions represented in sensor data, such as weather conditions (e.g., sunny, foggy, rainy, snowy, dusty), weather severity levels (e.g., 0 to 4), road or surface conditions (e.g., wet, damp, icy, snowy, dry), road or surface condition severity levels (e.g., 0 to 4), and / or some combination thereof (e.g., categories representing different combinations of weather conditions, weather severity levels, road or surface conditions, and / or road or surface condition severity levels). As an example of some possible combination categories, if there are five supporting weather conditions, five supporting weather severity levels, and five supporting road or surface conditions, there can be 5x5x5 = 125 combination categories, and the corresponding classifications that the weather and surface condition detection head can perform, and the combination category with the highest predictive probability can be selected.

[0032] In some embodiments, the extracted features (e.g., in a non-destructive view) can be applied to one or more output heads, such as an artifact detection head, which can predict the class label for each point based on any number of supported categories (e.g., solid objects or artifacts, their subclasses, etc.). For example, the artifact detection head can include any number of classification channels for each supported echo and can select the category with the highest prediction probability for each echo.

[0033] Data generated by the output head, or its representation (e.g., detected objects, airworthiness space, weather and / or road conditions, and / or artifacts), can be provided to the driving stack of an autonomous vehicle or other self-driving machine to enable safe planning and control of the autonomous vehicle. For example, the driving stack can use detected obstacles and detected airworthiness space to guide path planning and maneuvering decisions, can use detected weather and / or road conditions to guide speed and handling strategies (e.g., reducing speed on wet or icy roads) or monitor its Operational Design Domain (ODD) (e.g., ensuring the vehicle operates within predefined conditions for which it is designed and validated, adjusting operational behavior such as reducing speed in heavy rain or avoiding certain maneuvers on icy or wet roads, alerting the driver to take over control when conditions exceed the vehicle's safe operating parameters), and / or can use detected artifacts to filter out points that do not belong to real surfaces or objects to prevent false alarms (e.g., false braking due to incorrectly detected obstacles, false lane departure warnings due to misinterpreted road markings, false collision warnings due to incorrectly identified objects (such as overhanging branches or roadside debris).

[0034] Compared to existing technologies, this technology brings many improvements. For example, applying a lossless view to one or more channels of the DNN (which increases input resolution), evaluating sequential sensor data (e.g., LiDAR scans) to provide motion cues in any given inference, and / or projecting intermediate representations of the extracted features (instead of using explicit class supervision in the first stage) should all improve the final detection accuracy.

[0035] Furthermore, improved accuracy of the DNN output can be used to enhance one or more alternative classical detection algorithms. For example, in some scenarios, a conventional DNN may miss small obstacles (e.g., minor hazards such as debris, potholes, or uneven surfaces), or the accuracy of the detected shapes or surface structures may be limited (e.g., detected road surfaces). Therefore, one or more classical detection algorithms can be used downstream as backup algorithms to refine upstream results or catch potential missed detections. In some embodiments, data or representations generated by one or more output heads (e.g., detected objects, airworthiness spaces, weather and / or road conditions, artifacts, etc.) can be provided as input to one or more alternative classical detection algorithms to improve their performance. For example, after the neural network detects airworthiness spaces, a classical algorithm such as Random Sample Consensus (RANSAC) can be used to fit a road surface model (e.g., a plane) and detect minor hazards (e.g., outliers beyond a threshold distance from the fitted surface). Embodiments that improve the accuracy of detected airworthiness spaces can also improve the accuracy of the road surface model, the performance of downstream classical algorithms (e.g., by providing more accurate input data), and / or the accuracy of detected hazards. Improved road surface models and / or detected hazards can be used to enhance suspension control and / or avoid detected hazards.

[0036] In another example scenario (e.g., in calm weather), conventional DNNs might misinterpret weather artifacts such as steam from manhole covers or exhaust plumes as static hazards, leading to false braking or false collision warnings. In contrast, this technique can be used to detect such phenomena as artifacts (or improve their detection accuracy), designate them as outliers, and reduce their weights or exclude them during road surface model optimization, thereby avoiding misclassification as hazards and the resulting false braking and / or false collision warnings.

[0037] In some embodiments, detected weather and / or surface conditions can be used to improve control over a vehicle or other autonomous machine. For example, visibility optimization is a path planning process that attempts to maximize forward visibility and minimize obstructions (obstacles within the vehicle's critical line of sight) to enable earlier detection of potential obstacles. Classical visibility optimization techniques use weather or road classification to update the vehicle's route based on current environmental conditions. Embodiments that predict weather or surface conditions with improved accuracy can leverage these improved conditions to enhance visibility optimization and the corresponding path planning.

[0038] In another example scenario, a vehicle is traveling at highway speeds. If the vehicle encounters a dust plume blowing into the highway from the roadside, a classic detection algorithm might perceive the dust plume as an obstacle, and ADAS might apply emergency braking to avoid it. However, embodiments that improve artifact detection accuracy can filter out detected weather artifacts (such as dust) before performing object detection (or can filter out detection results corresponding to detected weather artifacts), thereby avoiding unnecessary braking that might be caused by existing techniques.

[0039] In some examples, machine learning models / DNNs can be packaged as microservices, such as inference microservices (e.g., NVIDIA NIM), which can include containers (e.g., operating system (OS) level virtualization packages) that may include an application programming interface (API) layer, a server layer, a runtime layer, and / or a model "engine." For example, an inference microservice may include the container itself and the model (e.g., weights and biases). In some cases, such as when the machine learning model / DNN is small enough (e.g., with a small enough number of parameters), the model may be included within the container itself. In other examples (e.g., when the model is large), the model may be hosted / stored in the cloud (e.g., in a data center) and / or may be hosted on-premises and / or at the edge (e.g., on a local server or computing device, but outside the container). In these embodiments, the model may be accessed via one or more APIs (e.g., a REST API). Therefore, in some embodiments, the machine learning models described herein can be deployed as inference microservices to accelerate model deployment on any cloud, data center, or edge computing system while ensuring data security. For example, an inference microservice may include one or more APIs, pre-configured containers for simplified deployment, an optimized inference engine (e.g., execution software built using standardized AI models, such as NVIDIA's Triton Inference Server), and / or one or more APIs for high-performance deep learning inference, which may include inference runtime and model optimizations to provide low latency and high throughput for production applications (e.g., NVIDIA's TensorRT), and / or enterprise management data for telemetry (e.g., including identity, metrics, health checks, and / or monitoring). The machine learning models described herein may be included as part of a microservice along with acceleration infrastructure capable of deployment using a single command and / or orchestration and autoscaling using a container orchestration system on the acceleration infrastructure (e.g., reaching data center scale on a single device). Therefore, an inference microservice may include a machine learning model (e.g., a model optimized for high-performance inference), inference runtime software for executing the machine learning model and providing output / response to inputs (e.g., user queries, prompts, etc.), and enterprise management software for providing health checks, identification, and / or other monitoring. In some embodiments, an inference microservice may include software for performing in-situ replacements and / or updates to the machine learning model. When replacing or updating, the software performing the replacement / update can maintain the user configurations of the inference runtime software and the enterprise management software.

[0040] Therefore, the techniques described herein can be used to detect and classify objects and / or parts or features of activities in the environment, and these detections and classifications can be provided to the autonomous vehicle driving stack to enable safety planning and control of autonomous vehicles.

[0041] refer to Figure 1 , Figure 1 This is an example detection pipeline 100 according to some embodiments of this disclosure. It should be understood that such and other arrangements described herein are merely illustrative examples. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groups, etc.) may be used in addition to or as alternatives to the arrangements and elements shown, and certain elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components, or in combination with other components, and can be implemented in any suitable combination and in any location. The various functions performed by the entities described herein can be performed by hardware, firmware, and / or software. For example, various functions can be implemented by a processor executing instructions stored in memory. In some embodiments, the systems, methods, and processes described herein can be used with… Figures 6A-6D The example autonomous vehicle shown is 600. Figure 7 The example computing device 700 and / or shown Figure 8 The example data center 800 shown is similar to other components, features, and / or functions used to perform this function.

[0042] exist Figure 1 In the illustrated example embodiment, the detection pipeline 100 includes a neural network 108 configured to detect objects, airworthiness space, weather or surface conditions, artifacts, and / or other parts or features of the environment based on sensor data 102 representing a 3D environment. The sensor data 102 may be preprocessed (e.g., via preprocessing 104) into input data 106 in a format supported by the neural network 108, and the input data 106 may be applied to the neural network 108 to generate one or more outputs, such as object data 110 representing a detected object 115, airworthiness space data 111 representing a detected airworthiness space 116, weather and surface condition data 112 representing detected weather or surface conditions 117, artifact data 113 representing detected artifacts 118, and / or other outputs representing other parts or features of the environment. The outputs may be provided to control components of the self-machine (e.g., controller 636, ADAS system 638, SOC 604, software stack 122, and / or...). Figures 6A-6D Other components of the autonomous vehicle 600 help the autonomous machine perform one or more operations in the environment (e.g., obstacle avoidance, path planning, mapping, navigation, control, etc.).

[0043] Typically, sensor data 102 can be generated using any number and type of sensors, such as, but not limited to, one or more cameras, LiDAR sensors, RADAR (radar) sensors, and / or other types of sensors (e.g., as described below). Figures 6A-6D (The sensors described in the autonomous vehicle 600). For example, sensor 101 may include one or more sensors 101 of the autonomous machine, such as LiDAR sensor 664, and sensor 101 may be used to generate sensor data 102 representing objects, surfaces, and / or other parts or features in the 3D environment surrounding the autonomous machine. Continuing with LiDAR data as an example, sensor data 102 may include raw sensor data, LiDAR point cloud data, and / or reflection data processed into some other format. For example, reflection data may be combined with position and orientation data (e.g., data from GNSS and IMU sensors) to form a point cloud representing detected environmental reflections. Each detection in the point cloud can identify the 3D location of that detection and metadata about that detection, such as one or more reflection characteristics. In some embodiments, the LiDAR sensor supports multiple echoes. Some non-limiting examples of LiDAR sensors include the Hesai P / AT series, Luminar Iris / Iris+ series, Velodyne HDL / VLS series, and Ouster OS1 / OS2 series LiDAR sensors.

[0044] Sensor data 102 can be preprocessed 104 into a format that the neural network 108 can understand. For example, in embodiments where sensor data 102 includes LiDAR data (and / or other data, such as RADAR data), the LiDAR data (and / or other data) can be accumulated, transformed to a single coordinate system (e.g., centered on the self-actuator / vehicle), self-motion compensated (e.g., compensated to the latest known position of the self-machine), and / or projected to form a projected image with specified spatial dimensions and pixel values ​​for any number of channels or layers storing corresponding reflectivity characteristics (e.g., azimuth, elevation, distance, intensity, Doppler velocity, reflectivity, signal-to-noise ratio, etc.), multiple echo characteristics, and / or other characteristics. Depending on the implementation, any suitable perspective projection (e.g., spherical, cylindrical, pinhole, orthogonal, non-destructive, etc.) can be used to generate any specified view of the environment (e.g., top view, front view, perspective view, etc.). For example, LiDAR point clouds can be projected to form LiDAR range images with (e.g., non-destructive) perspective, top-down or bird's-eye view (BEV) images, and / or other forms. Typically, the projected images and / or some other sensor data can be stored and / or encoded into any suitable representation (e.g., input data 106) that can be used as input to neural network 108.

[0045] At a higher level, neural network 108 can detect objects, airworthy spaces, weather or surface conditions, artifacts, and / or other parts or features of the environment represented in input data 106. For example, neural network 108 can extract object data 110, which includes classification data representing pixels belonging to certain categories of detected objects and / or instance regression data quantifying the location, shape, or orientation of detected objects, and can apply post-processing 114 (e.g., decoding) to generate representations of detected objects 115 (e.g., bounding boxes, closed polylines, or other bounding shapes used to identify the location, size, and / or orientation of detected objects; class labels; instance labels; distance data; etc.). In some embodiments, neural network 108 may extract airworthiness space data 111 (e.g., top-down occupancy grids) including categorical data indicating which parts of the environment (e.g., grid cells) are likely or predicted to be airworthiness (e.g., drivable or free space), and may apply any known post-processing 114 (e.g., smoothing, converting class confidences to binary classification values, and / or vectorizing paths) to generate a representation of detected airworthiness space 116. In some embodiments, neural network 108 may extract weather and surface condition data 112 including categorical data representing one or more weather and / or surface conditions detected in sensor data, such as weather conditions (e.g., sunny, foggy, rainy, snowy, dusty), weather severity levels (e.g., 0 to 4), road or surface conditions (e.g., wet, damp, icy, snowy, dry), road or surface condition severity levels (e.g., 0 to 4), and / or some combination thereof, and may apply post-processing 114 (e.g., selecting the class with the highest predictability) to generate a representation of detected weather or surface condition 117. In some embodiments, the neural network 108 may extract artifact data 113 including classification data that classifies each input raster cell and / or each encoded echo (e.g., represented by one or more corresponding depth-by-depth dimensions) in the input data 106 (e.g., as an entity or an artifact), and may apply post-processing 114 (e.g., selecting the category with the highest prediction probability) to generate a representation of the detected artifact 118.

[0046] In some embodiments, neural network 108 may be implemented using a DNN (e.g., a convolutional neural network (CNN)). While some embodiments describe the implementation of neural network 108 using neural networks (particularly CNNs), this is not intended to be limiting. For example, but not limited to, neural network 108 may include a variety of different networks or machine learning models of any type, such as machine learning models using linear regression, logistic regression, decision trees, support vector machines (SVM), Naive Bayes, k-nearest neighbors (Knn), K-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoders, convolutions, transformers, recurrent, perceptrons, long / short-term memory (LSTM), large language models (LLM), visual language models (VLM), multimodal language models, transformers, diffusion, encoder-only, decoder-only, encoder-decoder, Hopfield, Boltzmann, deep belief, deconvolution, generative adversarial, liquid machines, etc.) and / or other types of machine learning models.

[0047] Typically, the neural network 108 may include: one or more input heads (or at least partially discrete laminar flows) for processing different inputs represented by input data 106; one or more common backbones (or laminar flows) from which one or more input or output branches originate; and one or more output heads for predicting different outputs, any of which may include one or more feature extractors (e.g., DNN, encoder, decoder, etc.) including convolutional layers, pooling layers, and / or other layer types.

[0048] In some embodiments, the neural network 108 can be used to process multiple projected images (e.g., represented as corresponding tensors) that represent multiple views of sensor data 102 (e.g., LiDAR data) and / or multiple time slices (e.g., lossless LiDAR distance images representing multiple time slices). For example, the detection pipeline 100 can run at any suitable frame rate, generating one frame of sensor data 102 for each time slice, generating any number of projected images representing the corresponding views of that frame of sensor data 102, and applying the projected images to the neural network 108. Additionally or alternatively, the frame of sensor data 102, the projected images, and / or extracted features of the current time slice can be cached, retrieved, and applied to the neural network 108 in a subsequent time slice. In this way, the neural network 108 can evaluate multiple views of sensor data 102 and / or motion cues represented by sensor data 102 in different time slices (e.g., projected images corresponding to sequential LiDAR scans).

[0049] Figure 2AThis is an illustration of an example implementation of a neural network 108 according to some embodiments of the present disclosure, which can be used to process multiple views of sensor data 102 from multiple time slices. In this example, the LiDAR point cloud 205 representing the current time slice (t=0) is projected into two different views (e.g., via...). Figure 1 Preprocessing 104, namely top-down view (e.g., top-down projected image 210) and perspective view (e.g., non-destructive LiDAR distance image 220). Figure 2B This is an enlarged illustration of an example of lossless LiDAR distance. In this example, LiDAR point clouds 230 and 240, representing previous time slices (t-1 and tn, respectively), are (or have been) projected into a perspective view (e.g., lossless LiDAR distance images 235 and 245, respectively). Therefore, the top-down projection image 210 representing the current time slice, the lossless LiDAR distance image 220 representing the current time slice, the lossless LiDAR distance image 235 representing the previous time slice t-1, and the lossless LiDAR distance image 245 representing the previous time slice tn can be applied to a neural network 108 that processes these input data to generate object data 110, airworthiness space data 111, weather and surface condition data 112, and / or artifact data 113. This is merely an example, and other variations can be considered within the scope of this disclosure.

[0050] Figure 3This is an illustration of an example architecture of a neural network 108 according to some embodiments of the present disclosure. For example, the neural network 108 may include: a first stage (e.g., composed of input heads including corresponding feature extractors 330 and 340a-c) that processes multiple 2D views of a 3D environment; a second stage (e.g., composed of an encoder / decoder backbone 370) that processes extracted features in a common (e.g., top-down) 2D view (e.g., feature stack 360); and any number of output heads derived from the first or second stage (e.g., object detection head 375, airworthiness space detection head 380, weather / surface condition detection head 385, artifact detection head 390). In this example, the neural network 108 includes multiple input heads or channels for processing different views of sensor data (e.g., first view 310 and second view 320). Additionally or alternatively, neural network 108 may include multiple input heads or channels for processing sensor data from multiple time slices (e.g., second views 320a-n) to provide motion cues. Extracted features (e.g., learned intermediate representations, such as latent spatial tensors) extracted for different time slices (e.g., by corresponding feature extractors 340a-n) can be geometrically projected from second view 320 (e.g., geometric projection 350a-n) to first view 310, and can be combined (e.g., indicated by a plus sign) for projected extracted features for different time slices using any suitable technique (e.g., summation, averaging, concatenation, fusion via neural networks, etc.). The combined projected extracted features from different time slices can be combined with features extracted in first view 310 (e.g., by feature extractor 330) using any known technique to generate a combined representation of the extracted features in a common view (e.g., feature stack 360), which can be applied to encoder / decoder backbone 370.

[0051] In some embodiments, the encoder / decoder backbone 370 can be implemented using encoder and decoder components with skip connections (e.g., similar to feature pyramid networks, U-Net, etc.). For example, the encoder / decoder backbone 370 can accept a feature stack 360 (e.g., an input tensor) and apply various convolutions, pooling, and / or other types of operations to extract features into a latent space. In an example implementation, the encoder / decoder backbone 370 includes an encoding (shrinking) path and a decoding (expanding) path. Along the shrinking path, each resolution can include any number of layers (e.g., convolutions, dilated convolutions, inception blocks, etc.) and downsampling operations (e.g., max pooling). Along the expanding path, each resolution can include any number of layers (e.g., deconvolution, upsampling followed by convolution, and / or other types of operations). In the expanding path, each resolution of the feature map can be upsampled and concatenated with feature maps of the same resolution from the shrinking path (e.g., in the depth dimension). The corresponding resolutions of the shrinking and expanding paths can be connected by skips, which can be used to add or concatenate feature maps from the corresponding resolutions. Therefore, the encoder / decoder backbone 370 can extract features into a latent space tensor, which can be input to the object detection head 375, the airworthiness space detection head 380, the weather / surface condition detection head 385, and / or other output heads (e.g., in an embodiment where the artifact detection head 390 originates from the encoder / decoder backbone 370).

[0052] Object detection head 375 may include any number of layers (e.g., convolution, pooling, classifiers such as softmax, and / or other types of operations) for predicting object data 110, including categorical data, based on the output of encoder / decoder backbone 370. For example, object detection head 375 may include a channel (e.g., laminar flow plus a classifier) ​​for each category of the object or other part of the environment to be detected (e.g., vehicle, car, truck, vulnerable road user, pedestrian, cyclist, motorcycle, sidewalk, building, tree, telephone pole, its subclasses, some combination thereof), enabling object detection head 375 to extract categorical data in any suitable form. For example, object detection head 375 may predict a confidence map representing the inferred confidence level of a particular object (regardless of its category), or it may predict a separate confidence map for each category, and so on. In some embodiments, the category confidence data predicted by object detection head 375 may be in the form of a multi-channel tensor, where each channel can be considered a heatmap storing categorical values ​​(e.g., probability, score, or logit) representing the likelihood that each pixel belongs to the category corresponding to that channel.

[0053] Additionally or alternatively, the object detection head 375 may include any number of layers (e.g., convolution, pooling, classifiers such as softmax, and / or other types of operations) for predicting object data 110, including object instance data (e.g., the location, geometry, and / or orientation of detected objects), based on the output of the encoder / decoder backbone 370. The object detection head 375 may include any number of channels (e.g., laminar flow plus a classifier), wherein each channel regresses a specific type of information about the detected object instance, such as the object's location (e.g., a dx / dy vector pointing to the center or corner of the object), object height, object width, object orientation (e.g., rotation angle, such as sine and / or cosine), certain statistical measures of it (e.g., minimum, maximum, mean, median, variance, etc.), and / or the like. As a non-limiting example, the object detection head 375 may include separate dimensions: an x-dimensional dimension for identifying points (e.g., corners, centroids, etc.) of the detected object; a y-dimensional dimension for the points of the detected object; the width of the detected object; the height of the detected object; the sine of the orientation of the detected object (e.g., a rotation angle in 2D image space); the cosine of the orientation of the detected object; and / or other types of information. These types of object instance data are intended only as examples, and other types of object information may be additionally or alternatively regressed and / or otherwise predicted. The object detection head 375 may include a separate regression channel for each category, or a set of channels for all categories. In some embodiments, the instance regression data predicted by the object detection head 375 may be in the form of a multi-channel tensor, wherein each channel may include a floating-point number regressing a specific type of object information (e.g., a specific object dimension).

[0054] Airworthiness space detection head 380 may include any number of layers (e.g., convolutions, pooling, classifiers such as softmax, and / or other types of operations) for predicting airworthiness space data 111 based on the output of encoder / decoder backbone 370. For example, airworthiness space detection head 380 may include channels (e.g., laminar flow plus a classifier) ​​that extract classification data that categorizes each grid cell (e.g., in a top-down occupancy grid) as potentially occupied (e.g., occupied by obstacles) or airworthiness (e.g., drivable or free space, no obstacles, etc.). Typically, airworthiness space detection head 380 may generate any number of classification scores for each grid cell (e.g., one for occupied classification and one for airworthiness classification) so that airworthiness space detection head 380 can predict confidence maps representing inferred confidence levels indicating whether a particular grid cell is occupied or airworthiness, separate confidence maps for each category, and so on. In some embodiments, the category confidence data predicted by the airworthiness space detection head 380 may be in the form of a (e.g., multi-channel) top-down occupancy grid, where one channel may be viewed as a heatmap storing a classification value (e.g., probability, score, or logit) representing the likelihood that each grid cell is occupied, and / or one channel represents the likelihood that each grid cell represents airworthiness space.

[0055] The weather / surface condition detection head 385 may include any number of layers (e.g., convolution, pooling, classifiers such as softmax, and / or other types of operations) for predicting weather and surface condition data 112, including categorical data, based on the output of the encoder / decoder backbone 370. For example, the weather / surface condition detection head 385 may include a channel (e.g., laminar flow plus a classifier) ​​for each category of weather and / or surface condition to be detected (e.g., weather condition, weather severity level, road or surface condition, road or surface condition severity level, or some combination thereof, such as categories representing different combinations of weather condition, weather severity level, road or surface condition, and / or road or surface condition severity level), such that the weather / surface condition detection head 385 extracts categorical data in any suitable form. For example, the weather / surface condition detection head 385 may predict a confidence score for each supported category of weather or surface condition, which represents the inferred confidence level of the presence or absence of the corresponding category of weather or surface condition. In some embodiments, the category confidence data predicted by the weather / surface condition detection head 385 may be in the form of a one-dimensional vector, where different positions in the vector represent the corresponding categories.

[0056] In some embodiments, extracted features extracted in one or more views (e.g., by feature extractor 330, feature extractor 340a) can be applied to one or more output heads, such as artifact detection head 390 and / or other output heads (e.g., in...). Figure 3 In other embodiments not shown, there is an object detection head 375, an airworthiness space detection head 380, and / or a weather / surface condition detection head 385. Figure 3 In the illustrated embodiment, features extracted from the second view 320a (e.g., a non-destructive LiDAR distance image) are applied to the artifact detection head 390. In some embodiments, features extracted from the second view 320a from multiple time slices (e.g., extracted by feature extractors 340a-n) can be combined and applied to the artifact detection head 390 using any suitable technique.

[0057] The artifact detection head 390 may include any number of layers (e.g., convolution, pooling, classifiers such as softmax, and / or other types of operations) for predicting artifact data 113, including categorical data, based on the output of the encoder / decoder backbone 370. For example, the second view 320a may store features of multiple echoes in corresponding layers, and thus may include multiple layers corresponding to multiple encoded attributes and for each echo supported by the LiDAR sensor (e.g., three attributes of each of three echoes can be encoded in nine layers), for the maximum expected number of points projectable onto a pixel, and / or other cases. Therefore, the second view 320a may align sensor data (e.g., detected 3D LiDAR points) with a 2D grid, where each cell or pixel includes multiple layers for each supported attribute and each supported echo (e.g., three attributes multiplied by three echoes). Thus, the artifact detection head 390 may include a channel (e.g., laminar flow plus a classifier) ​​for each supported echo so that the artifact detection head 390 extracts categorical data in any suitable form. For example, the artifact detection head 390 can predict a confidence map for each echo channel, representing the inferred confidence level of whether the LiDAR echo projected onto the corresponding cell or pixel is an artifact (e.g., a weather artifact, a sensor artifact), and can also predict separate confidence maps for different categories of artifacts, and so on. In some embodiments, the category confidence data predicted by the artifact detection head 390 can be in the form of a multi-channel tensor, where each channel can be viewed as a heatmap storing classification values ​​(e.g., probability, score, or logit) representing the likelihood that the LiDAR point corresponding to each pixel and / or echo channel represents an artifact.

[0058] Notice, Figure 3The architecture shown is intended as an example only, and various variations can be implemented within the scope of this disclosure (e.g., input or output headers originating from different locations).

[0059] Figure 4 This is an illustration of an example implementation of a neural network 108 according to some embodiments of the present disclosure, using cached features 420a-n from previous time slices. In some embodiments, the neural network 108 may process sensor data (e.g., projected images representing sequential LiDAR scans) sequentially at a specified frame rate, and the extracted features may be cached and used for multiple inferences. Figure 4 In the example shown, neural network 108 includes input channels that accept corresponding projected images representing corresponding time slices (e.g., three input channels for three consecutive projected images (e.g., lossless LiDAR distance images in second views 320a-n)). For any given time slice, feature extractor 340a can extract features from the second view 320a of that time slice, which can be cached for subsequent inference and used for current inference. For example, cached features 420a-n previously extracted by feature extractor 340a for one or more previous time slices can be retrieved and aligned with the extracted features of the current time slice by performing self-motion compensation on the cached features 420a-n of previous time slices to reflect the current position and orientation of the sensor (e.g., transforming previously detected distance values ​​to compensate for the updated position and orientation of the ego machine), and the extracted features of the current time slice and previous time slices can be geometrically projected (e.g., geometric projection 350a-n) into a common coordinate system.

[0060] generally, Figure 1 , Figure 2A , Figure 3 and / or Figure 4Training data for the neural network 108 can be acquired and / or generated in various ways. For example, input training data can be generated from sensor data using the techniques described herein for operating the neural network 108. Corresponding ground truth training data can be generated in various ways. In some embodiments, the sensor data and / or the corresponding input training data (e.g., projected images) can be annotated with labels or other markers identifying the location, geometry, orientation, and / or category of instances of relevant objects and / or other environmental parts represented in the sensor data (e.g., manually, automatically, etc.). In some examples, labels can be generated in drawing programs (e.g., annotation programs), computer-aided design (CAD) programs, annotation programs, other suitable programs of other types, and / or can be hand-drawn. In any example, labels can be synthetically generated (e.g., generated from computer models or renderings), realistically generated (e.g., designed and generated based on real-world data), machine-automated (e.g., extracting features from data using feature analysis and learning, and then generating labels), manually annotated (e.g., an annotator or annotation expert defining the location of the labels), and / or a combination of the above (e.g., manually identifying the vertices of polylines, and the machine generating polygons using a polygon rasterizer). Typically, labels may include bounding boxes, closed polylines, or drawings, annotations, overlays, and / or other bounding shapes associated with sensor data.

[0061] For example, in some embodiments, ground truth data for object detection head 375 can be derived from the bounding shape of annotated objects (e.g., using the position, geometry, orientation, and / or category of each annotation to generate a corresponding ground truth segmentation mask that matches the view, size, and dimensions of the output of object detection head 375). Additionally or alternatively, any known technique can be used to detect and / or regress objects of any specified category of 2D or 3D shape represented in the input training data (e.g., point cloud segmentation, projecting a 3D point cloud onto a 2D view and then evaluating the resulting 2D projected image, e.g., as described in U.S. Patent No. 11,532,168), and the corresponding ground truth data for object detection head 375 can be derived from the detected objects.

[0062] In some embodiments, ground truth data for the airworthiness space detection head 380 can be derived from annotated road boundaries or road boundaries acquired during data collection via a map application programming interface (API), as well as annotated obstacles and / or hazards (e.g., 2D or 3D bounding boxes). For example, the data collection vehicle can access a map service via the API, which provides road boundaries in response to geographic location-based queries. In some embodiments where the airworthiness space detection head 380 operates in a top-down view, annotations can be generated in the top-down view (e.g., an occupancy grid) or projected onto the top-down view, and cells in the top-down view can be marked as occupied or drivable using ray projection. For example, rays can be projected in different directions from a reference point (e.g., vehicle location) within the top-down view (e.g., an occupancy grid), and each ray can continue to extend until it intersects an annotated obstacle (e.g., represented by an annotated object) or an annotated road boundary. Areas where the rays travel without hitting any obstacles or boundaries can be marked as ground truth airworthiness space. Additionally or alternatively, truth airworthiness space can be directly annotated (e.g., manually).

[0063] The ground truth data used for the weather / surface condition detection head 385 can be obtained from one or more weather service providers during data collection. For example, the data collection vehicle can access a weather service provider (such as OpenWeatherMap) via API, which provides real-time weather data in response to geographic location-based queries, such as weather type (e.g., rain, snow, fog), weather severity level (e.g., light, moderate, heavy), road conditions (e.g., dry, wet, icy), and road condition severity (e.g., slight, heavy, severe). The weather data acquired within a given time slice can be correlated with the corresponding sensor data within that time slice, converted to the format output by the weather / surface condition detection head 385, and used as the corresponding ground truth data.

[0064] The ground truth data used for the artifact detection head 390 can be derived from the input LiDAR data or other detected 3D points. For example, any known technique can be used to filter out points belonging to dynamic objects and identify the remaining non-static points in the scene, which may belong to weather or road conditions, such as rain or snow in the air or on the ground. For example, LiDAR points surrounded by annotated or detected objects belonging to the moving object category can be identified as belonging to dynamic objects. LiDAR points surrounded by annotated static obstacles can be identified as belonging to static objects. Therefore, the remaining LiDAR points can be classified as part of the non-static scene and labeled as artifacts. In some embodiments, points identified as part of dynamic objects can be subtracted from the points identified as non-static, and the remaining points can be labeled as artifacts. Therefore, artifact labels can be converted to the format output by the artifact detection head 390 and used as the corresponding ground truth data.

[0065] Therefore, the neural network 108 can be trained using training data, any suitable training technique, and any suitable loss function.

[0066] Now back Figure 1 In some embodiments, post-processing 114 may be applied, for example, to decode one or more outputs of neural network 108, convert the outputs into a format acceptable to one or more downstream components, and / or perform other operations. For example, post-processing 114 may be applied to object data 110 (e.g., decoding, filtering, clustering, deduplication, etc.) to generate a representation of detected objects 115 (e.g., bounding boxes, closed polylines, or other bounding shapes identifying the location, size, and / or orientation of detected objects; class labels; instance labels; distance data; etc.); post-processing 114 may be applied to airworthiness space data 111 (e.g., smoothing, converting class confidence or segmented images into vectorized paths) to generate a representation of detected airworthiness space 116; post-processing 114 may be applied to weather and surface condition data 112 (e.g., selecting the class with the highest predictability) to generate a representation of detected weather or surface condition 117; and / or post-processing 114 may be applied to artifact data 113 (e.g., selecting the class with the highest predictability) to generate a representation of detected artifacts 118. In some embodiments, post-processing 114 may include filtering out detected objects 115 formed by detected artifacts 118, or updating the bounding box, closed polyline, or other bounding shape of the detected objects 115 to omit portions corresponding to the detected artifacts 118. In some embodiments, the output of neural network 108 and / or its post-processed representation may be used as output 121 and provided to control components of the self-machine (e.g., controller 636, ADAS system 638, SOC 604, software stack 122, and / or...). Figures 6A-6DOther components of the autonomous vehicle 600 help the autonomous machine perform one or more operations within the environment (e.g., obstacle avoidance, path planning, mapping, navigation, control, etc.).

[0067] In some embodiments, one or more classical detection algorithms 120 may be used for redundancy and / or as backups to refine the output (or representation thereof) of neural network 108, or to detect objects or other features that neural network 108 may miss. For example, classical detection algorithm 120 may use classical computer vision to perform object detection based on sensor data 102 (e.g., using a low-level LiDAR perception stack that does not use DNN and is executed in parallel with neural network 108), and post-processing 114 may include merging common detections and including the union of objects detected by the two systems in output 121. In some embodiments, classical detection algorithm 120 may apply classical ground plane or surface estimation techniques (e.g., RANSAC) to fit a road surface model (e.g., a plane), distinguish minor hazards (e.g., outliers beyond a threshold distance from the fitted surface) from airworthiness space (e.g., other points on the fitted surface), and generate a representation of airworthiness space, and post-processing 114 may include merging the representations of the detected airworthiness space detected by the two systems and including the result in output 121. These are merely examples, and other types of classical detection algorithms 120 may be implemented within this disclosure.

[0068] In some embodiments, one or more outputs of the neural network 108 may be used to improve the performance and / or accuracy of the classical detection algorithm 120. For example, representations of airworthiness spaces detected by the neural network 108 (e.g., airworthiness space data 111, detected airworthiness space 116) and / or representations of detected artifacts detected by the neural network 108 (e.g., artifact data 113, detected artifact 118) may be provided to the classical detection algorithm 120, and a road surface model (e.g., a plane) may be fitted to the airworthiness space detected by the neural network 108 using classical ground or surface estimation techniques (e.g., RANSAC). Artifacts detected by the neural network 108 may be deweighted or excluded during road surface model optimization, outliers exceeding a threshold distance from the fitted surface may be identified as hazards, and the road surface model and / or detected hazards may be included in the output 121. Embodiments that improve the accuracy of detected airworthiness space and / or detected artifacts can also improve the accuracy of road surface models, the performance of downstream classical detection algorithm 120 (e.g., by providing more accurate input data) and / or the accuracy of detected hazards.

[0069] Therefore, the output 121 can be obtained from Figures 6A-6DThe control components of the depicted autonomous vehicle 600 utilize, for example, an autonomous driving software stack 122 executed on one or more components of the vehicle 600 (e.g., SoC 1004, CPU 1018, GPU 1020, accelerator, etc.). For example, the vehicle 600 can use detected obstacles and detected airworthy space to guide path planning and maneuvering decisions, can use detected weather and / or road conditions to guide speed and handling strategies (e.g., slowing down on wet or icy roads) for path planning (e.g., as input for visibility optimization) and / or monitor its Operational Design Domain (ODD) (e.g., ensuring the vehicle operates under predefined conditions designed and validated for it, adjusting operational behavior (e.g., slowing down in heavy rain or avoiding certain maneuvers on icy or slippery roads), alerting the driver to take over control when conditions exceed the vehicle's safe operating parameters), and / or can use detected artifacts to filter out points that do not belong to real surfaces or objects to prevent false positives.

[0070] In some embodiments, output 121 may be used by one or more layers of the autonomous driving software stack 122 (which may alternatively be referred to herein as "driving stack 122"). Driving stack 122 may include a sensor manager (not shown), perception components (e.g., perception layers corresponding to driving stack 122), a world model manager 126, a planning component 128 (e.g., planning layers corresponding to driving stack 122), a control component 130 (e.g., control layers corresponding to driving stack 122), an obstacle avoidance component 132 (e.g., obstacle avoidance or collision avoidance layers corresponding to driving stack 122), an actuation component 134 (e.g., actuation layers corresponding to driving stack 122), and / or other components corresponding to additional and / or alternative layers of driving stack 122. In some examples, detection pipeline 100 may be performed by the perception component, which may feed the various layers of driving stack 122 up to the world model manager, as described in more detail herein.

[0071] The sensor manager can manage and / or abstract sensor data 102 from the vehicle's 600 sensors. For example, refer to... Figure 6CSensor data 102 can be generated by LiDAR sensor 664 (e.g., continuously, intermittently, or based on certain conditions). The sensor manager can receive sensor data 102 in different formats from the sensors (e.g., sensors of the same type may output sensor data in different formats) and can be configured to convert different formats to a uniform format (e.g., for each sensor of the same type). Therefore, other components, features, and / or functions of the autonomous vehicle 600 can use the uniform format, thereby simplifying the processing of sensor data 102. In some examples, the sensor manager can use the uniform format to apply control back to the sensors of vehicle 600, such as to set the frame rate or perform gain control. The sensor manager can also use timestamps to update sensor packets or communications corresponding to the sensor data to help inform the various components, features, and functions of the autonomous vehicle control system about the processing of the sensor data.

[0072] The world model manager 126 can be used to generate, update, and / or define a world model. The world model manager 126 can use information generated by and received from the perception components of the driving stack 122 (e.g., the location of detected obstacles). Perception components may include obstacle sensors, path sensors, waiting sensors, map sensors, and / or other perception components. For example, the world model can be defined at least in part based on the availability of obstacles, paths, and waiting conditions, which can be perceived in real-time or near real-time by obstacle sensors, path sensors, waiting sensors, and / or map sensors. The world model manager 126 can continuously update the world model based on newly generated and / or received inputs (e.g., data) from obstacle sensors, path sensors, waiting sensors, map sensors, and / or other components of the autonomous vehicle control system.

[0073] The world model can be used to help inform the planning component 128, control component 130, obstacle avoidance component 132, and / or actuation component 134 of the driving stack 122. An obstacle sensor can perform obstacle perception based on the location where the vehicle 600 is permitted or capable of driving (e.g., based on the location of a drivable path or other navigable path defined by avoiding detected obstacles), and the speed at which the vehicle 600 can drive without colliding with obstacles (e.g., objects, such as structures, entities, vehicles, etc.) sensed by the vehicle 600's sensors and / or neural network 108.

[0074] A path perception device can perform path perception, for example, by perceiving nominal paths available under specific conditions. In some examples, the path perception device can also consider lane changes for path perception. A lane map can represent one or more paths available to vehicle 600 and can be as simple as a single path on a highway entrance ramp. In some examples, a lane map can include paths to the desired lane and / or can indicate lane changes available along the highway (or other road type), or can include nearby lanes, lane changes, intersections, turns, cloverleaf interchanges, mergings, and / or other information.

[0075] Waiting sensors can be responsible for determining restrictions on vehicle 600 based on rules, conventions, and / or practical considerations. For example, rules, conventions, and / or practical considerations may relate to traffic lights, multi-way stopping, yielding, lane changing, toll booths, gates, police or other emergency personnel, road workers, stopped buses or other vehicles, one-way bridge arbitration, ferry entrances, etc. Therefore, waiting sensors can be used to identify potential obstacles and implement one or more controls (e.g., deceleration, stopping, etc.) that might be impossible with obstacle sensors alone.

[0076] Map sensors can include mechanisms that identify behaviors and, in some examples, determine specific instances of which conventions should be applied in a particular area. For example, a map sensor can determine, based on data representing previous driving or trips: that no U-turns occurred at an intersection during certain times of day; that electronic signs displaying lane directionality change according to the time of day; that two traffic lights that are close together (e.g., almost offset from each other) are associated with different roads; that in Rhode Island, the first vehicle waiting to turn left at a traffic light violated the law by turning before oncoming traffic when the light turned green; and / or other information. Map sensors can inform vehicle 600 of static or stationary infrastructure objects and obstacles. Map sensors can also generate information for waiting sensors and / or route sensors, for example, to determine which light at an intersection must be green for vehicle 600 to take a specific route.

[0077] In some examples, information from the map sensor can be sent, transmitted, and / or provided to the server (e.g., Figure 6DThe map manager of server 678 can send, transmit, and / or provide information from the server to the map sensor and / or location manager of vehicle 600. The map manager may include a cloud map application located at a remote location of vehicle 600, and vehicle 600 may access the cloud map application via one or more networks. For example, the map sensor and / or location manager of vehicle 600 may communicate with one or more other components or features of the map manager and / or server to inform the map sensor and / or location manager of vehicle 600's past and current driving or trips, as well as the past and current driving or trips of other vehicles. The map manager can provide map output (e.g., map data) that can be located by the location manager based on a specific location of vehicle 600, and the world model manager 126 can use the located map output to generate and / or update a world model.

[0078] Planning component 128 may include a route planner, a lane planner, a behavior planner, and a behavior selector, as well as other components, features, and / or functions. The route planner may generate a planned path using information from a map sensor, a map manager, and / or a positioning manager, along with other information. This planned path may consist of GNSS waypoints (e.g., GPS waypoints) and 3D world coordinates (e.g., Cartesian coordinates, polar coordinates, etc.), which indicate coordinates relative to the origin on vehicle 600. Waypoints may represent specific distances from vehicle 600 to the future, such as city blocks, kilometers, feet, inches, miles, etc., which can be used as targets for the lane planner.

[0079] The lane planner can use a lane map (e.g., a lane map from a path aggregator), object poses within the lane map (e.g., according to a positioning manager), and / or a target point and direction at that future distance from the route planner as input. The target point and direction can be mapped to the best-matching drivable point and direction in the lane map (e.g., based on GNSS and / or compass orientation). A graph search algorithm can then be performed on the lane map, starting from the current edge, to find the shortest path to the target point.

[0080] The behavior planner can determine the feasibility of basic behaviors of vehicle 600 (e.g., staying in the lane or changing lanes left or right) in order to match feasible behaviors with the most desired behaviors output by the lane planner. For example, if the desired behavior is determined to be unsafe and / or unavailable, a default behavior can be selected instead (e.g., the default behavior could be staying in the lane when the desired behavior or changing lanes is unsafe).

[0081] Control component 130 can follow the trajectory or path (lateral and longitudinal) received from the behavior selector of planning component 128 as closely as possible within the capabilities of vehicle 600 (e.g., based on output 121, such as detected objects and detected airworthiness space). Control component 130 can use tight feedback to handle unmodeled, unplanned events or behaviors and / or any factors that cause a deviation from the ideal situation (e.g., unexpected delays). In some examples, control component 130 can use a forward predictive model that takes control as an input variable and produces predictions that can be compared to the desired state (e.g., compared to the desired lateral and longitudinal paths requested by planning component 128). One or more controls that minimize the deviation can be determined.

[0082] Although planning component 128 and control component 130 are shown separately, this is not intended to be limiting. For example, in some embodiments, the boundary between planning component 128 and control component 130 may not be precisely defined. Therefore, at least some components, features, and / or functions belonging to planning component 128 may be associated with control component 130, and vice versa. This also applies to any component shown separately in the drive stack 122.

[0083] Obstacle avoidance component 132 helps autonomous vehicle 600 avoid collisions with objects (e.g., moving and stationary objects). Obstacle avoidance component 132 may include a "primitive" obstacle avoidance computing mechanism and may act as the "survival brain" or "crawling brain" of vehicle 600. In some examples, obstacle avoidance component 132 may be used independently of components, features, and / or functions in vehicle 600 required to comply with traffic rules and civilized driving. In these examples, the obstacle avoidance component may disregard traffic regulations, road rules, and civilized driving norms to ensure that vehicle 600 does not collide with any objects. Therefore, the obstacle avoidance layer may be a separate layer from the road rules layer, and the obstacle avoidance layer may ensure that vehicle 600 performs safe operations solely from an obstacle avoidance perspective. On the other hand, the road rules layer may ensure that the vehicle complies with traffic regulations and conventions and adheres to legal and regular right-of-way (as described herein).

[0084] In some examples, obstacle avoidance component 132 may use output 121 (e.g., airworthy space (or other airworthy path) and / or detected objects) to determine the controls or actions to be taken. For example, a drivable path may provide indication to obstacle avoidance component 132, telling vehicle 600 where it can maneuver without colliding with any object, structure, and / or similar object, or at least where there is no static structure present.

[0085] In a non-limiting embodiment, obstacle avoidance component 132 may be implemented as a separate, discrete feature of vehicle 600. For example, obstacle avoidance component 132 may operate separately from the planning layer, control layer, actuation layer, and / or other layers of the driving stack 122 (e.g., in parallel, before, and / or after).

[0086] Therefore, vehicle 600 can use this information (e.g., as the edge or track of a path) to navigate, plan, or otherwise perform one or more actions in the environment (e.g., lane keeping, lane changing, merging, lane splitting, etc.).

[0087] While some embodiments use one or more outputs of neural network 108 or their representations (e.g., output 121) to control autonomous vehicles or other self-contained machines, this is not mandatory. For example, in some embodiments, one or more outputs of neural network 108 or their representations can be used to create training data. For instance, one or more frames of sensor data 102 may be specified or otherwise used as input training data, one or more outputs of neural network 108 (e.g., object data 110, airworthiness space data 111, surface condition data 112, artifact data 113) may be specified or otherwise used as corresponding ground truth training data, and any number of input and ground truth training data frames may be included in the training dataset. Thus, the training data can be used to train machine learning models (e.g., neural networks, such as neural networks that perform perception based on input image data, RADAR data, LiDAR data, ultrasonic data, and / or other types of data) to detect objects, airworthiness spaces, weather or surface conditions, artifacts, and / or other parts or features of the environment.

[0088] Now for reference Figure 5 Each block of the method 500 described herein includes a computational process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions can be implemented by a processor executing instructions stored in memory. Method 500 can also be embodied as computer-usable instructions stored on a computer storage medium. Method 500 can be provided by a standalone application, a standalone service, a managed service (alone or in combination with other managed services), or a plug-in to another product, etc. Furthermore, this document provides examples regarding... Figure 1 The detection pipeline 100 in the document describes method 500. However, this method, as well as other methods described herein, may be additionally or alternatively performed by any system or combination of systems, including but not limited to the systems described herein.

[0089] Figure 5This is a flowchart of a method 500 for multi-view LiDAR perception according to some embodiments of the present disclosure. Method 500 includes, at block B502: processing data representing LiDAR data generated using at least one LiDAR sensor employed by an autonomous machine using one or more neural networks (NNs) to generate first extracted feature data in a first view and second extracted feature data in a second view of the plurality of views. For example, regarding... Figure 1 In the detection pipeline 100, the neural network 108 can be used to process multiple projected images (e.g., represented as corresponding tensors) representing multiple views of sensor data 102 (e.g., LiDAR data). Reference Figure 3 The neural network 108 may include a first stage (e.g., consisting of an input head including corresponding feature extractors 330 and 340a-c) that processes multiple 2D views of a 3D environment, and the feature extractors 330 and 340a-c can extract corresponding features from different views.

[0090] Method 500 at block B504 includes: generating combined extracted feature data that combines first extracted feature data in a first view with a projected representation of second extracted feature data projected onto the first view. For example, regarding Figure 3 The extracted features in the second view 320a can be geometrically projected (via geometric projection 35a) into the first view 310, and can be combined with features extracted in the first view 310 (e.g. by feature extractor 330) using any known technique to generate a combined representation of the extracted features in the common view (e.g., feature stack 360).

[0091] Method 500, at block B506, includes: using one or more neural networks to process the combined extracted feature data to generate one or more outputs. For example, regarding... Figure 3 The feature stack 360 can be applied to the encoder / decoder backbone 370, and the encoder / decoder backbone 370 can extract features into a latent space tensor, which can be input to an object detection head 375, an airworthiness space detection head 380, a weather / surface condition detection head 385, and / or other output heads (e.g., in an embodiment where the artifact detection head 390 originates from the encoder / decoder backbone 370).

[0092] Method 500 at block B508 includes: controlling one or more operations of the ego machine based on at least one or more outputs. For example, regarding Figure 1The output of the detection pipeline 100, the neural network 108, and / or its post-processed representation can be used as output 121 and provided to the control components of the self-machine (e.g., controller 636, ADAS system 638, SOC 604, software stack 122, and / or...). Figures 6A-6D Other components of the autonomous vehicle 600 help the autonomous machine perform one or more operations in the environment (e.g., obstacle avoidance, path planning, mapping, etc.).

[0093] The systems and methods described herein may be used by, but are not limited to, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, aircraft, boats, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, engineering vehicles, trains, underwater vehicles, remotely controlled vehicles such as drones, and / or other vehicle types. Furthermore, the systems and methods described herein can be used for a variety of purposes, such as, but not limited to: machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, safety and supervision, simulation and digital twins, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital twins, data center processing, conversational AI, optical transport simulation (e.g., ray tracing, path tracing, etc.), generative AI applications, language model applications (e.g., large language model (LLM), visual language model (VLM), etc.), collaborative content creation of 3D assets, cloud computing and / or any other suitable applications.

[0094] The disclosed embodiments can be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines), systems implemented using robots, aviation systems, medical systems, boating systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems containing one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems for performing generative AI operations, systems implemented at least partially in a data center, systems for performing conversational AI operations, systems implementing one or more language models (e.g., one or more large language models (LLMs), visual language models (VLMs), one or more multimodal language models, etc.), systems for hosting real-time streaming applications, systems for presenting one or more of virtual reality content, augmented reality content, or mixed reality content, systems for performing optical transmission simulations, systems for performing collaborative content creation of 3D assets, systems implemented at least partially using cloud computing resources, and / or other types of systems.

[0095] In some embodiments, the systems and methods described herein can be performed in a simulated environment (e.g., NVIDIA's DriveSIM) using simulated data (e.g., simulated sensor data from simulated sensors of a virtual machine or simulated machine). For example, simulated sensor data (e.g., LiDAR data representing a simulated environment (e.g., a highway or warehouse environment) generated from the perspective of one or more virtual sensors of a simulated self-machine) and / or simulated projected images can be applied to a DNN to detect simulated objects, simulated airworthiness spaces, simulated weather or surface conditions, and / or simulated artifacts in the simulated environment, and the output of the DNN can be used to control the simulated self-machine in the environment. These simulation operations can be used to test the performance of underlying algorithms, systems, and / or processes before deploying them to the real world. In some cases, simulations can be used to generate synthetic training data (e.g., virtual sensor data and / or corresponding projected images representing a simulated environment and generated from the perspective of one or more virtual sensors of a simulated self-machine), and (in addition to or as a substitute for real-world data) synthetic training data can also be used to train the DNN. In any example, such as when using a simulated environment for testing, validation, training, etc., one or more optical transport algorithms (e.g., ray tracing and / or path tracing algorithms) may be used to render or otherwise generate the simulated environment and / or associated training data. In some embodiments, simulated environments and / or one or more objects, features, or components thereof may be generated or managed within a 3D content collaboration platform (e.g., NVIDIA's OMNIVERSE) for industrial digitization, generative physics AI, and / or other use cases, applications, or services. For example, the content collaboration platform or system may include a system that uses or develops generic scene descriptors (USD) (e.g., OpenUSD) data to manage objects, features, scenes, etc., in simulated environments, digital environments, etc. The platform may include realistic physics simulations, such as using NVIDIA's PhysX SDK, to simulate real physics and physical interactions with simulations hosted on the platform. This platform can integrate OpenUSD and ray tracing / path tracing / light transport simulations (such as NVIDIA's RTX rendering technology) into software tools and simulation workflows for building, training, deploying, or testing AI systems, such as systems for testing, validating, training (e.g., machine learning models, neural networks, etc.) and / or other tasks related to automobiles, robots, machines, or other applications.

[0096] Example autonomous vehicles

[0097] Figure 6AThis is an illustration of an example autonomous or semi-autonomous vehicle or machine 600 according to some embodiments of the present disclosure. The autonomous or semi-autonomous vehicle or machine 600 (alternately referred to herein as “vehicle 600”, “machine 600”, “self-vehicle 600”, “self-machine 600”, “robot 600”, etc.) may include, but is not limited to, passenger vehicles such as cars, trucks, buses, first-response vehicles, shuttles, electric or motorized bicycles, motorcycles, fire trucks, police cars, ambulances, boats, engineering vehicles, underwater vehicles, robotic vehicles, drones, aircraft, vehicles coupled to trailers (e.g., semi-trailer trucks for hauling cargo) and / or other types of vehicles (e.g., driverless and / or vehicles accommodating one or more passengers). Autonomous vehicles are typically described according to their level of automation, as defined by the National Highway Traffic Safety Administration (NHTSA) of the U.S. Department of Transportation, and by the Society of Automotive Engineers (SAE) in its "Classification and Definition of Terms Related to Driving Automation Systems for Road Motor Vehicles" (Standard No.: J3016-201806, published June 15, 2018; Standard No.: J3016-201609, published September 30, 2016; and previous and future versions of this standard). Vehicle 600 may implement functions according to one or more of Level 3 to Level 5 of autonomous driving. Vehicle 600 may be able to have one or more functionalities that conform to Level 1 to Level 5 of autonomous driving. For example, according to an embodiment, vehicle 600 may have driver assistance (Level 1), partial automation (Level 2), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5). As used herein, the term “autonomy” can include any and / or all types of autonomy of the vehicle 600 or other machines, such as full autonomy, high autonomy, conditional autonomy, partial autonomy, provision of auxiliary autonomy, semi-autonomy, primary autonomy or other specified autonomy.

[0098] Vehicle 600 may include a chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other vehicle components. Vehicle 600 may include a propulsion system 650, such as an internal combustion engine, a hybrid power plant, an all-electric motor, and / or other propulsion system types. Propulsion system 650 may be connected to the drivetrain of vehicle 600, which may include a transmission to allow propulsion of vehicle 600. Propulsion system 650 may be controlled in response to receiving signals from throttle valve / accelerator 652.

[0099] When the propulsion system 650 is in operation (e.g., when the vehicle is moving), the steering system 654, including a steering wheel, can be used to guide the vehicle 600 (e.g., along a desired path or route). The steering system 654 can receive signals from the steering actuator 656. For fully automatic (level 5) functionality, the steering wheel may be optional.

[0100] The brake sensor system 646 can be used to operate the vehicle brakes in response to signals received from the brake actuator 648 and / or the brake sensor.

[0101] One or more controllers 636 may include one or more system-on-chip (SoC) 604 ( Figure 6C One or more controllers and / or one or more GPUs may provide signals (e.g., representing commands) to one or more components and / or systems of vehicle 600. For example, one or more controllers may send signals via one or more brake actuators 648 to operate vehicle brakes, via one or more steering actuators 656 to operate steering system 654, and via one or more throttles / accelerators 652 to operate propulsion system 650. One or more controllers 636 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operating commands (e.g., signals representing commands) to allow autonomous driving and / or assist a human driver in driving vehicle 600. One or more controllers 636 may include a first controller 636 for autonomous driving functions, a second controller 636 for functional safety functions, a third controller 636 for artificial intelligence functions (e.g., computer vision), a fourth controller 636 for infotainment functions, a fifth controller 636 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 636 may handle two or more of the functions described above, and two or more controllers 636 may handle a single function and / or any combination thereof.

[0102] One or more controllers 636 may provide signals for controlling one or more components and / or systems of vehicle 600 in response to sensor data (e.g., sensor inputs) received from one or more sensors. Sensor data may be received from, for example, but not limited to, one or more Global Navigation Satellite System (“GNSS”) sensors 658 (e.g., one or more Global Positioning System sensors), one or more radar sensors 660, one or more ultrasonic sensors 662, one or more lidar sensors 664, one or more inertial measurement unit (IMU) sensors 666 (e.g., one or more accelerometers, one or more gyroscopes, one or more magnetometers, magnetometers, etc.), one or more microphones 696, one or more stereo cameras 668, one or more wide-angle lenses, etc. Camera 670 (e.g., fisheye camera), one or more infrared cameras 672, one or more surround cameras 674 (e.g., 360-degree cameras), one or more long-range and / or mid-range cameras 698, one or more speed sensors 644 (e.g., for measuring the speed of vehicle 600), one or more vibration sensors 642, one or more steering sensors 640, one or more brake sensors (e.g., as part of brake sensor system 646), one or more occupant monitoring system (OMS) sensors 601 (e.g., one or more interior cameras) and / or other sensor types.

[0103] One or more of the controllers 636 may receive inputs (e.g., represented by input data) from the instrument cluster 632 of the vehicle 600 and provide outputs (e.g., represented by output data, displayed data, etc.) via a human-machine interface (HMI) display 634, an audio signaler, a speaker, etc., and / or via other components of the vehicle 600. Outputs may include information such as vehicle speed, rate, time, map data (e.g., ...). Figure 6C Information such as a high-definition (“HD”) map 622, location data (e.g., the location of vehicle 600, such as its location on a map), direction, the location of other vehicles (e.g., grid occupancy), and information about objects and their states perceived by one or more controllers 636. For example, the HMI display 634 may display information about the presence of one or more objects (e.g., street signs, warning signs, traffic light changes, etc.), and / or information about driving actions that the vehicle has performed, is performing, or will perform (e.g., changing lanes now, exiting from exit 34B in two miles, etc.).

[0104] Vehicle 600 also includes a network interface 624, which can communicate over one or more networks using one or more wireless antennas 626 and / or a modem. For example, network interface 624 may be able to communicate via Long Term Evolution (“LTE”), Wideband Code Division Multiple Access (“WCDMA”), Universal Mobile Telecommunications System (“UMTS”), Global System for Mobile Communications (“GSM”), IMT-CDMA Multicarrier (“CDMA2000”), etc. One or more wireless antennas 626 may also allow communication between objects in the environment (e.g., vehicles, mobile devices, etc.) using one or more local area networks (e.g., Bluetooth, Bluetooth Low Energy (“LE”), Z-Wave, ZigBee, etc.) and / or Low Power Wide Area Networks (“LPWAN”), such as LoRaWAN, SigFox, etc.).

[0105] Figure 6B According to some embodiments of this disclosure Figure 6A An example of the camera position and field of view of an exemplary autonomous vehicle 600. The camera and its respective field of view are an example embodiment and are not intended to be limiting. For example, additional and / or alternative cameras may be included and / or the cameras may be located at different positions on the vehicle 600.

[0106] The camera type may include, but is not limited to, a digital camera, which may be suitable for components and / or systems of vehicle 600. One or more cameras may operate at Automotive Safety Integrity Level (ASIL) B and / or other ASILs. According to embodiments, the camera type may have any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc. The camera may use a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include a red transparent (RCCC) color filter array, a red transparent blue (RCCB) color filter array, a red blue green transparent (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or other types of color filter arrays. In some embodiments, a sharp-pixel camera, such as a camera with RCCC, RCCB, and / or RBGC color filter arrays, may be used to improve light sensitivity.

[0107] In some examples, one or more cameras can be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundancy or fail-safe design). For example, a multi-functional single camera can be installed to provide functions such as lane departure warning, traffic sign assistance, and intelligent headlight control. One or more cameras (e.g., all cameras) can simultaneously record and provide image data (e.g., video).

[0108] One or more cameras may be mounted in mounting components, such as custom-designed (3D-printed) components, to cut off stray light and interior reflections (e.g., dashboard reflections from the windshield rearview mirror) that could interfere with the camera's ability to capture image data. Regarding wing mirror mounting components, these components may be custom-3D printed so that the camera mounting plate matches the shape of the wing mirror. In some examples, one or more cameras may be integrated into the wing-shaped rearview mirror. For side-view cameras, one or more cameras may also be integrated into the four pillars at each corner of the cab.

[0109] A camera with a field of view including a portion of the environment in front of the vehicle (e.g., a front-facing camera) can be used for surround view to help identify forward paths and obstacles, and, with the assistance of one or more controllers 636 and / or control SOCs, to provide information crucial for generating an occupancy grid and / or determining the preferred vehicle path. The front-facing camera can be used to perform many of the same ADAS functions as lidar, including emergency braking, pedestrian detection, and collision avoidance. The forward-facing camera can also be used in ADAS functions and systems, including lane departure warning (“LDW”), adaptive cruise control (“ACC”), and / or other functions such as traffic sign recognition.

[0110] Various cameras can be used in front-facing configurations, including, for example, monocular camera platforms that include a complementary metal-oxide-semiconductor (“CMOS”) color imager. Another example could be a wide-angle camera 670 that can be used to perceive objects entering the field of view from the periphery (e.g., pedestrians, cross traffic, or bicycles). Although Figure 6B Only one wide-angle camera is shown, but the vehicle 600 may have any number (including zero) of wide-angle cameras 670. Furthermore, any number of remote cameras 698 (e.g., a pair of long-angle stereo cameras) can be used for depth-based object detection, particularly for objects for which neural networks have not yet been trained. One or more remote cameras 698 can also be used for object detection and classification, as well as basic object tracking.

[0111] Any number of stereo cameras 668 may also be included in the front-mounted configuration. In at least one embodiment, one or more stereo cameras 668 may include an integrated control unit comprising a scalable processing unit that may provide programmable logic (“FPGA”) and a multi-core microprocessor with an integrated controller area network (“CAN”) or Ethernet interface on a single chip. This unit can be used to generate a 3D map of the vehicle environment, including distance estimates for all points in the image. One or more alternative stereo cameras 668 may include a compact stereo vision sensor that may include two camera lenses (one on each side) and an image processing chip that measures the distance from the vehicle to a target object and uses the generated information (e.g., metadata) to activate automatic emergency braking and lane departure warning functions. In addition to the stereo cameras described herein, or alternatively, other types of stereo cameras 668 may be used.

[0112] A camera with a field of view including a portion of the vehicle's side environment (e.g., a side-view camera) can be used in the surround view to provide information for creating and updating the occupancy mesh and generating side collision warnings. For example, one or more surround cameras 674 (e.g., such as...) Figure 6B The four surround cameras 674 shown may be positioned on the vehicle 600. One or more surround cameras 674 may include one or more wide-angle cameras 670, one or more fisheye cameras, one or more 360-degree cameras, etc. For example, four fisheye cameras may be located at the front, rear, and sides of the vehicle. In an alternative arrangement, the vehicle may use three surround cameras 674 (e.g., left, right, and rear) and may utilize one or more other cameras (e.g., a front-facing camera) as a fourth surround-view camera.

[0113] Cameras with a view that includes a portion of the environment behind the vehicle 600 (e.g., rear-view cameras) can be used for parking assistance, surround view, rear-end collision warning, and creating and updating occupancy grids. A variety of cameras can be used, including but not limited to cameras that are also suitable as front-facing cameras (e.g., one or more long-range and / or mid-range cameras 698, one or more stereo cameras 668, one or more infrared cameras 672, etc.), as described herein.

[0114] Cameras (e.g., one or more OMS sensors 601) covering a portion of the interior environment of the vehicle's 600-seat cabin can be used as part of an Occupant Monitoring System (OMS), such as, but not limited to, a Driver Monitoring System (DMS). For example, the OMS sensors (e.g., OMS sensor 601) can be used (e.g., by controller 636) to track the gaze direction, head posture, and / or blinking of occupants and / or drivers. This gaze information can be used to determine the level of attention of the occupant or driver (e.g., detecting drowsiness, fatigue, and / or distraction), and / or to take responsive measures to prevent harm to the occupant or operator. In some embodiments, the data from the OMS sensors can be used to implement gaze control operations triggered by the driver and / or non-driver passengers, such as (but not limited to) adjusting cabin temperature and / or airflow, opening and closing windows, controlling cabin lighting, controlling the entertainment system, adjusting rearview mirrors, adjusting seat position, and / or other operations. In some embodiments, the OMS can be used for applications such as determining when an object and / or passenger is left in the cabin (e.g., detecting the presence of a passenger after the driver has alighted).

[0115] Figure 6C According to some embodiments of this disclosure Figure 6A A block diagram of an exemplary system architecture for an exemplary autonomous vehicle 600 is provided. It should be understood that this and other arrangements described herein are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groups, etc.) may be used in addition to the arrangements and elements shown, or other arrangements and elements may be used instead of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components, or in combination with other components, and can be implemented in any suitable combination and location. The various functions described herein as being performed by entities can be performed by hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory.

[0116] Figure 6C Every component, feature, and system of vehicle 600 is connected via bus 602. Bus 602 may include a Controller Area Network (CAN) data interface (also referred to herein as the "CAN bus"). CAN can be a network within vehicle 600 used to help control various features and functions of vehicle 600, such as the actuation of brakes, acceleration, braking, steering, windshield wipers, etc. The CAN bus can be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., a CAN ID). The CAN bus can be read to locate steering wheel angle, ground speed, engine speed per minute (RPM), button positions, and / or other vehicle status indicators. The CAN bus may conform to the ASIL B standard.

[0117] Although bus 602 is described herein as a CAN bus, this is not intended to be limiting. For example, in addition to or from a CAN bus, FlexRay and / or Ethernet may also be used. Furthermore, although bus 602 is represented by a single line, this is not intended to be limiting. For example, any number of buses 602 may exist, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 602 may be used to perform different functions and / or for redundancy. For example, a first bus 602 may be used for a collision avoidance function, and a second bus 602 may be used for drive control. In any example, each bus 602 may communicate with any component of vehicle 600, and two or more buses 602 may communicate with the same component. In some examples, each SoC 604, each controller 636, and / or each computer within the vehicle may access the same input data (e.g., input from sensors of vehicle 600) and may be connected to a common bus, such as a CAN bus.

[0118] Vehicle 600 may include one or more controllers 636, as described herein. Figure 6A The controller 636 is described above. Controller 636 can be used for various functions. One or more controllers 636 can be coupled to any of the various other components and systems of vehicle 600, and can be used to control vehicle 600, artificial intelligence of vehicle 600, infotainment of vehicle 600, etc.

[0119] Vehicle 600 may include one or more System-on-Chip (SoC) 604. SoC 604 may include one or more CPUs 606, one or more GPUs 608, one or more processors 610, one or more caches 612, one or more accelerators 614, one or more data storage 616, and / or other components and features not shown. One or more SoCs 604 can be used to control vehicle 600 in various platforms and systems. For example, one or more SoCs 604 may be combined with an HD map 622 in a system (e.g., the system of vehicle 600), the HD map 622 being accessible via a network interface 624 from one or more servers (e.g., [server name missing]). Figure 6D Server 678) receives map refresh and / or updates.

[0120] One or more CPUs 606 may include CPU clusters or CPU complexes (or referred to herein as “CCPLEX”). One or more CPUs 606 may include multiple cores and / or a L2 cache. For example, in some embodiments, one or more CPUs 606 may include eight cores in a coherent multiprocessor configuration. In some embodiments, one or more CPUs 606 may include four dual-core clusters, each with a dedicated L2 cache (e.g., 2MB L2 cache). One or more CPUs 606 (e.g., CCPLEX) may be configured to support simultaneous cluster operation, such that any combination of clusters of CPUs 606 is active at any given time.

[0121] One or more CPU 606s can implement power management capabilities including one or more of the following features: automatic clock gating of a single hardware block when idle to conserve dynamic power; clock gating of each core when the core is not actively executing instructions due to executing WFI / WFE instructions; independent power gating of each core; independent clock gating of each core cluster when all cores are clock-gated or power-gated; and / or independent power gating of each core cluster when all cores are power-gated. One or more CPU 606s can further implement enhanced algorithms for managing power states, specifying allowed power states and expected wake-up times, and the hardware / microcode determines the optimal power state for the core, cluster, and CCPLEX to enter. The processing core can support simplified power state input sequences in software and offload the work to the microcode.

[0122] One or more GPUs 608 may include integrated GPUs (or referred to herein as “iGPUs”). GPUs 608 may be programmable and efficient for parallel workloads. In some examples, one or more GPUs 608 may use an enhanced tensor instruction set. One or more GPUs 608 may include one or more streaming microprocessors, wherein each streaming microprocessor may include a Level 1 cache (e.g., a Level 1 cache with at least 96KB of storage), and two or more streaming microprocessors may share a Level 2 cache (e.g., a Level 2 cache with 512KB of storage). In some embodiments, one or more GPUs 608 may include at least eight streaming microprocessors. One or more GPUs 608 may use one or more computation application programming interfaces (APIs). Furthermore, one or more GPUs 608 may use one or more parallel computing platforms and / or programming models (e.g., NVIDIA’s CUDA).

[0123] One or more GPU 608s can be power-optimized for optimal performance in automotive and embedded use cases. For example, one or more GPU 608s can be fabricated on FinFETs. However, this is not intended to limit, and other semiconductor manufacturing processes can be used to fabricate one or more GPU 608s. Each streaming microprocessor can combine multiple mixed-precision processing cores partitioned into multiple blocks. For example, but not limited to, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block could be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA tensor cores for deep learning matrix algorithms, an L0 instruction cache, a thread bundle scheduler, a dispatch unit, and / or a 64KB register file. Furthermore, the streaming microprocessor can include independent parallel integer and floating-point data paths to provide efficient execution of workloads through mixed computation and addressing computation. The streaming microprocessor can include independent thread scheduling capabilities to allow for finer-grained synchronization and cooperation between parallel threads. Streaming microprocessors can include a combination of a level-one data cache and a shared memory unit to improve performance while simplifying programming.

[0124] One or more GPUs 608 may include a high-bandwidth memory (HBM) and / or a 16GB HBM2 memory subsystem to provide a peak memory bandwidth of approximately 900GB / s in some examples. In some examples, in addition to HBM memory, or optionally from HBM memory, synchronous graphics random access memory (SGRAM), such as fifth-generation graphics dual data rate synchronous random access memory (GDDR5), may be used.

[0125] The fifth-generation GPU 608 may include unified memory technology, which includes access counters to allow more accurate migration of memory pages to the processors that access them most frequently, thereby improving the efficiency of shared memory ranges between processors. In some examples, address translation service (ATS) support may be used to allow one or more GPUs 608 to directly access the page tables of one or more CPUs 606. In such examples, when one or more GPUs 608 memory management units (MMUs) experience a miss, an address translation request may be sent to one or more CPUs 606. In response, one or more CPUs 606 may look up the virtual-to-physical mapping of the address in their page tables and send the translation back to one or more GPUs 608. Therefore, unified memory technology allows for a single unified virtual address space for the memory of both one or more CPUs 606 and one or more GPUs 608, thereby simplifying the programming of one or more GPUs 608 and porting applications to one or more GPUs 608.

[0126] In addition, one or more GPUs 608 may include access counters that track the frequency with which one or more GPUs 608 access the memory of other processors. Access counters help ensure that memory pages are moved to the physical memory of the processor that accesses those pages most frequently.

[0127] One or more SoCs 604 may include any number of caches 612, including the caches 612 described herein. For example, one or more caches 612 may include an L3 cache available to one or more CPUs 606 and one or more GPUs 608 (e.g., connecting both one or more CPUs 606 and one or more GPUs 608). Caches 612 may include write-back caches with traceable thread state, for example, by using cache coherence protocols (e.g., MEI, MESI, MSI, etc.). Although a small cache size may be used, according to embodiments, the L3 cache may include 4 MB or more.

[0128] One or more SoCs 604 may include one or more arithmetic logic units (ALUs) that can be used to perform processing related to various tasks or operations of the vehicle 600, such as processing a DNN. Furthermore, one or more SoCs 604 may include one or more floating-point units (FPUs) or other mathematical coprocessors or digital coprocessor types for performing mathematical operations within the system. For example, one or more SoCs 604 may include one or more FPUs integrated as execution units within a CPU 606 and / or a GPU 608.

[0129] One or more SoCs 604 may include one or more accelerators 614 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, one or more SoCs 604 may include a hardware acceleration cluster that may include optimized hardware accelerators and / or large on-chip memory. Large on-chip memory (e.g., 4MB of SRAM) can allow the hardware acceleration cluster to accelerate neural networks and other computations. The hardware acceleration cluster can be used to supplement one or more GPUs 608 and offload some tasks from one or more GPUs 608 (e.g., freeing up more cycles of one or more GPUs 608 to perform other tasks). For example, one or more accelerators 614 may be sufficiently stable to be suitable for accelerating target workloads (e.g., perception, convolutional neural networks (CNNs), etc.). The term "CNN" as used herein may include all types of CNNs, including region-based or region-based convolutional neural networks (RCNNs) and fast RCNNs (e.g., for object detection).

[0130] One or more accelerators 614 (e.g., hardware acceleration clusters) may include one or more deep learning accelerators (DLAs). One or more DLAs may include one or more tensor processing units (TPUs) configured to provide an additional trillion operations per second for deep learning applications and inference. TPUs may be accelerators configured to perform image processing functions (e.g., for CNNs, RCNNs, etc.) and optimized for them. One or more DLAs may also be optimized for specific neural network types and floating-point operations and inference. One or more DLAs are designed to provide higher performance per millimeter than general-purpose GPUs and significantly outperform CPUs. One or more TPUs may perform multiple functions, including single-instance convolution functions, such as supporting INT8, INT16, and FP16 data types for features and weights, and post-processor functions.

[0131] One or more DLAs can execute neural networks, especially CNNs, quickly and efficiently on processed or unprocessed data for any of a variety of functions, including but not limited to: CNNs for object recognition and detection using data from camera sensors; CNNs for distance estimation using data from camera sensors; CNNs for emergency vehicle detection and recognition using microphone data; CNNs for facial recognition and vehicle owner recognition using data from camera sensors; and / or CNNs for safety and / or safety-related events.

[0132] One or more DLAs can perform any function of one or more GPUs 608. For example, by using inference accelerators, designers can perform any function for one or more DLAs or one or more GPUs 608. For example, designers can centralize the processing of CNNs and floating-point operations on one or more DLAs and leave other functions to one or more GPUs 608 and / or one or more other accelerators 614.

[0133] One or more accelerators 614 (e.g., hardware acceleration clusters) may include programmable vision accelerators (PVAs), which may also be referred to herein as computer vision accelerators. One or more PVAs may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. One or more PVAs may provide a balance between performance and flexibility. For example, each PVA may include, for example, but not limited to, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors.

[0134] The RISC core can interact with an image sensor (e.g., the image sensor of any camera described herein), one or more image signal processors, etc. Each RISC core may include any amount of memory. The RISC core can use any of a variety of protocols, depending on the implementation. In some examples, the RISC core can run a real-time operating system (RTOS). The RISC core can be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, the RISC core may include an instruction cache and / or tightly coupled RAM.

[0135] DMA allows PVA components to access system memory independently of one or more CPUs 606. DMA can support any features used to optimize PVA, including but not limited to support for multidimensional addressing and / or circular addressing. In some examples, DMA can support addressing in up to six or more dimensions, which may include block width, block height, block depth, horizontal block step, vertical block step, and / or depth step.

[0136] Vector processors can be programmable processors designed to efficiently and flexibly execute computer vision algorithms and provide signal processing capabilities. In some examples, a PVA may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem may operate as the main processing engine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or a vector memory (e.g., a VMEM). The VPU core may include a digital signal processor, such as a single-instruction, multiple-data (SIMD) or very long instruction word (VLIW) digital signal processor. The combination of SIMD and VLIW can improve throughput and speed.

[0137] Each vector processor may include an instruction cache and may be coupled to dedicated memory. As a result, in some examples, each vector processor may be configured to execute independently of other vector processors. In other examples, vector processors included in a particular PVA may be configured to employ data parallelism. For example, in some embodiments, multiple vector processors included in a single PVA may execute the same computer vision algorithm on different regions of an image. In other examples, vector processors included in a particular PVA may execute different computer vision algorithms simultaneously on the same image, or even different algorithms on a sequence of images or portions of an image. Among other things, any number of PVAs may be included in a hardware-accelerated cluster, and any number of vector processors may be included in each PVA. Furthermore, PVAs may include additional error-correcting code (ECC) memory to enhance overall system security.

[0138] One or more accelerators 614 (e.g., a hardware acceleration cluster) may include on-chip computer vision network and SRAM for providing high-bandwidth, low-latency SRAM for one or more accelerators 614. In some examples, the on-chip memory may include at least 4 MB of SRAM, including but not limited to eight field-configurable memory blocks accessible by the PVA and DLA. Each pair of memory blocks may include an Advanced Peripheral Bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory can be used. The PVA and DLA access memory via a backbone that provides high-speed memory access for the PVA and DLA. The backbone may include an on-chip computer vision network that interconnects the PVA and DLA to memory (e.g., using an APB).

[0139] An on-chip computer vision network may include an interface that determines that both the PVA and DLA have provided ready and valid signals before transmitting any control signals / addresses / data. Such an interface can provide independent phases and independent channels for transmitting control signals / addresses / data, as well as burst communication for continuous data transmission. This type of interface may conform to ISO 26262 or IEC 61508 standards, but other standards and protocols may also be used.

[0140] In some examples, one or more SoCs 604 may include a real-time ray tracing hardware accelerator, as described in U.S. Patent No. 10,885,698, issued January 5, 2021. The real-time ray tracing hardware accelerator can be used to rapidly and efficiently determine the location and extent of an object (e.g., within a world model), generate real-time visualization simulations for radar signal interpretation, sound propagation synthesis and / or analysis, sonar system simulation, general wave propagation simulation, comparison with lidar data for localization and / or other functions and / or uses. In some embodiments, one or more tree traversal units (TTUs) may be used to perform one or more ray tracing-related operations.

[0141] One or more accelerators (e.g., hardware accelerator clusters) have wide applications in autonomous driving. A PVA (Programmable Vision accelerator) could be used in critical processing stages of ADA (Advanced Driver Assistance Systems) and autonomous vehicles. The capabilities of a PVA are well-suited for algorithmic domains requiring predictable processing with low power consumption and low latency. Accordingly, PVAs perform well on semi-intensive or conventionally intensive computations, even on small datasets that require predictable runtimes with low latency and low power consumption. Therefore, in the context of autonomous vehicle platforms, PVAs are designed to run classic computer vision algorithms, as they are highly efficient in object detection and integer mathematical operations.

[0142] For example, according to one embodiment of this technology, a PVA is used to perform computer stereo vision. In some examples, a semi-global matching-based algorithm can be used, although this is not intended to limit it. Many applications of Level 3-5 autonomous driving require instantaneous motion estimation / stereo matching (e.g., motion structures, pedestrian recognition, lane detection, etc.). A PVA can perform computer stereo vision functions on input from two monocular cameras.

[0143] In some examples, PVA can be used to perform dense optical flow, providing processed radar data based on the raw radar data (e.g., using 4D Fast Fourier Transform). In other examples, PVA is used for time-of-flight depth processing, for example, providing processed time-of-flight data by processing the raw time-of-flight data.

[0144] DLA can be used to run any type of network to enhance control and driving safety, including neural networks that output a confidence measure for each object detection. Such a confidence value can be interpreted as a probability or to provide a relative “weight” for each detection relative to other detections. This confidence value allows the system to further determine which detections should be considered true positives rather than false positives. For example, the system can set a confidence threshold and only consider detections exceeding the threshold as true positives. In an Automatic Emergency Braking (AEB) system, false positives would cause the vehicle to automatically perform emergency braking, which is clearly undesirable. Therefore, only the most reliable detections should be considered as triggers for AEB. DLA can run a neural network to regress the confidence value. The neural network can take at least a subset of parameters as its input, such as bounding box dimensions, obtained ground plane estimates (e.g., from another subsystem), inertial measurement unit (IMU) sensor 666 outputs related to the vehicle's 600-degree orientation and distance, and three-dimensional position estimates of objects obtained from the neural network and / or other sensors (e.g., lidar sensor 664 or radar sensor 660).

[0145] One or more SoCs 604 may include one or more data storage units 616 (e.g., memory). The data storage unit 616 may be on-chip memory of the SoC 604, which may store neural networks to be executed on the GPU and / or DLA. In some examples, the capacity of the data storage unit 616 may be large enough to store multiple neural network instances for redundancy and security. The data storage unit 616 may include a level 2 or level 3 cache 612. As described herein, references to one or more data storage units 616 may include references to memory associated with the PVA, DLA, and / or one or more other accelerators 614.

[0146] One or more SoCs 604 may include one or more processors 610 (e.g., embedded processors). Processor 610 may include a boot and power management processor, which may be a dedicated processor and subsystem for handling boot power and management functions, as well as related security implementations. The boot and power management processor may be part of a boot sequence for one or more SoCs 604s and may provide runtime power management services. The boot power and management processor may provide clock and voltage programming, system low-power state transition assistance, management of SoC 604 thermal sensors and temperature sensors, and / or management of SoC 604 power states. Each temperature sensor may be implemented as a ring oscillator with an output frequency proportional to temperature, and one or more SoCs 604s may use the ring oscillator to detect the temperature of one or more CPUs 606s, one or more GPUs 608s, and / or one or more accelerators 614s. If a temperature is determined to exceed a threshold, the boot and power management processor may enter a temperature fault routine and place one or more SoCs 604s into a low-power state and / or place vehicle 600 into a driver-safe parking mode (e.g., safely parking vehicle 600).

[0147] One or more processors 610 may also include a set of embedded processors that can serve as an audio processing engine. The audio processing engine may be an audio subsystem capable of providing full hardware support for multi-channel audio through multiple interfaces and a wide range of flexible audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core of a digital signal processor with dedicated RAM.

[0148] One or more processors 610 may also include a normally-on processor engine that provides the necessary hardware functionality to support low-power sensor management and wake-up use cases. The normally-on processor engine may include a processor core, tightly coupled RAM, peripheral support (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0149] One or more processors 610 may also include a secure cluster engine, which includes a dedicated processor subsystem for handling security management for automotive applications. The secure cluster engine may include two or more processor cores, tightly coupled RAM, support for peripheral devices (e.g., timers, interrupt controllers, etc.), and / or routing logic. In secure mode, the two or more cores may operate in lockstep mode and function as a single core with comparison logic to detect any differences in their operations.

[0150] One or more processors 610 may also include a real-time camera engine, which may include a dedicated processor subsystem for handling real-time camera management.

[0151] One or more processors 610 may also include a high dynamic range signal processor, which may include an image signal processor, which is a hardware engine as part of the camera processing pipeline.

[0152] One or more processors 610 may include a video image synthesizer, which may be a processing block (e.g., implemented on a microprocessor) that implements the video post-processing functions required by the video playback application to generate the final image of the player window. The video image synthesizer may perform lens distortion correction on one or more wide-angle cameras 670, one or more surround cameras 674, and / or in-cabin monitoring camera sensors. The in-cabin monitoring camera sensors are preferably monitored by a neural network running on another instance of an advanced SoC, configured to recognize in-cabin events and respond accordingly. The in-cabin system may perform lip reading to activate cellular service and make phone calls, dictate emails, change vehicle destination, activate or change the vehicle's infotainment system and settings, or provide voice-activated web browsing. Some functions are only available to the driver when the vehicle is running in automatic mode; otherwise, they are disabled.

[0153] Video image synthesizers may include enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, in the case of motion in a video, noise reduction appropriately weights spatial information, thereby reducing the weight of information provided by adjacent frames. In cases where an image or part of an image does not contain motion, temporal noise reduction performed by the video image synthesizer can use information from the previous image to reduce noise in the current image.

[0154] The video image compositor can also be configured to perform stereoscopic correction on input stereoscopic shot frames. When the operating system desktop is in use, the video image compositor can also be used for user interface compositing, without requiring the GPU 608 to continuously render new surfaces. Even when one or more GPUs 608 are powered on and active during 3D rendering, the video image compositor can be used to offload one or more GPUs 608 to improve performance and responsiveness.

[0155] One or more SoCs 604 may also include a Mobile Industrial Processor Interface (MIPI) camera serial interface, a high-speed interface, and / or a video input block that can be used for receiving video and input from a camera and associated pixel input functions. One or more SoCs 604 may also include one or more input / output controllers that may be software-controlled and can be used to receive I / O signals that are not assigned a specific role.

[0156] One or more SoCs 604 may also include a wide range of peripheral interfaces to allow communication with peripheral devices, audio codecs, power management and / or other devices. One or more SoCs 604 may be used to process data from cameras (e.g., via gigabit multimedia serial links and Ethernet connections), sensors (e.g., one or more LiDAR sensors 664, one or more radar sensors 660, etc., connected via Ethernet), from bus 602 (e.g., vehicle 600 speed, steering wheel position, etc.), and from one or more GNSS sensors 658 (e.g., via Ethernet or CAN bus connections). One or more SoCs 604 may also include a dedicated high-performance, high-capacity memory controller, which may include its own DMA engine and may be used to free one or more CPUs 606 from routine data management tasks.

[0157] One or more SoCs 604 can form an end-to-end platform with a flexible architecture spanning automation levels 3-5, providing a comprehensive functional safety architecture that leverages and effectively utilizes computer vision and ADAS technologies to achieve diversity and redundancy, providing a platform for a flexible and reliable driver software stack and deep learning tools. Compared to traditional systems, one or more SoCs 604 can be faster, more reliable, and even more energy-efficient and space-saving. For example, when one or more accelerators 614 are combined with one or more CPUs 606, one or more GPUs 608, and one or more data storage units 616, a fast and efficient platform can be provided for Level 3-5 autonomous vehicles.

[0158] Therefore, this technology offers capabilities and functionalities that are unavailable in traditional systems. For example, computer vision algorithms can be executed on a CPU, which can be configured using a high-level programming language (such as C) to execute a wide variety of processing algorithms on diverse visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. In particular, many CPUs cannot execute complex object detection algorithms in real time, which is a requirement for automotive ADAS applications and practical Level 3-5 autonomous vehicles.

[0159] Compared to traditional systems, the techniques described in this paper, by providing CPU complexes, GPU complexes, and hardware acceleration clusters, allow for the simultaneous and / or sequential execution of multiple neural networks and the combination of results to enable Level 3–5 autonomous driving capabilities. For example, a CNN executing on a DLA or dGPU (e.g., one or more GPU 620s) can include text and character recognition, allowing a supercomputer to read and understand traffic signs, including signs for which the neural network has not yet been specifically trained. The DLA can also include a neural network capable of recognizing, interpreting, and providing semantic understanding of symbols, and passing this semantic understanding to a path planning module running on the CPU complex.

[0160] Another example is the ability to run multiple neural networks simultaneously, as required for Level 3, 4, or 5 driving. For instance, a warning sign consisting of "Warning: Flashing lights indicate icing conditions" and a light can be interpreted independently or jointly by multiple neural networks. The sign itself can be recognized as a traffic sign by a first deployed neural network (e.g., a trained neural network), while the text "Flashing lights indicate icing conditions" can be interpreted by a second deployed neural network, which, when a flashing light is detected, notifies the vehicle routing software (preferably executed on a CPU complex) of the presence of icing conditions. A third deployed neural network can be used to identify the flashing light and notify the vehicle routing software of its presence by operating it across multiple frames. All three neural networks can run simultaneously, for example, within a DLA and / or on one or more GPUs 608.

[0161] In some examples, the CNN used for facial recognition and owner identification can use data from camera sensors to identify the presence of an authorized driver and / or the owner of the vehicle 600. The engine can be unlocked using a normally open sensor when the owner approaches the driver's door and turns on the lights, and the vehicle can be disabled in safe mode when the owner leaves. In this way, one or more SoCs 604 provide anti-theft and / or carjacking protection.

[0162] In another example, the CNN for emergency vehicle detection and identification can use data from microphone 696 to detect and identify emergency vehicle sirens. Unlike conventional systems that use a general classifier to detect sirens and manually extract features, one or more SoCs(s) 604 use the CNN to classify environmental and urban sounds, as well as visual data. In a preferred embodiment, the CNN running on the DLA is trained to identify the relative closing speed of emergency vehicles (e.g., by using the Doppler effect). The CNN can also be trained to identify emergency vehicles specific to the vehicle's operating area, as identified by one or more GNSS sensors 658. Thus, for example, when operating in Europe, the CNN will seek to detect European sirens, and when operating in the United States, the CNN will seek to identify only North American sirens. Once an emergency vehicle is detected, an emergency vehicle safety routine can be executed using a control program with the aid of ultrasonic sensors 662, causing the vehicle to slow down, pull over, stop, and / or idle until one or more emergency vehicles pass.

[0163] The vehicle may include one or more CPUs 618 (e.g., one or more discrete CPUs or one or more dCPUs) coupled to one or more SoCs 604 via high-speed interconnects (e.g., PCIe). For example, one or more CPUs 618 may include x86 processors. The CPUs 618 can be used to perform any of a variety of functions, including arbitrating potentially inconsistent results between ADAS sensors and SoCs 604, and / or monitoring the status and health of one or more controllers 636 and / or infotainment SoCs 630.

[0164] Vehicle 600 may include one or more GPUs 620 (e.g., one or more discrete GPUs or one or more dGPUs) coupled to SoC 604 via high-speed interconnects (e.g., NVIDIA's NVLINK). One or more GPUs 620 may provide additional artificial intelligence capabilities, such as by executing redundant and / or different neural networks, and may be used to train and / or update neural networks based on inputs from sensors of vehicle 600 (e.g., sensor data).

[0165] Vehicle 600 may also include a network interface 624, which may include one or more wireless antennas 626 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). Network interface 624 can be used to enable wireless connectivity via the Internet to the cloud (e.g., with one or more servers 678 and / or other network devices), other vehicles, and / or computing devices (e.g., a passenger's client device). For communication with other vehicles, direct and / or indirect links can be established between the two vehicles (e.g., via a network and the Internet). A vehicle-to-vehicle communication link can provide a direct link. A vehicle-to-vehicle communication link can provide vehicle 600 with information about vehicles nearby (e.g., vehicles in front, to the side, and / or behind vehicle 600). This functionality may be part of vehicle 600's cooperative adaptive cruise control function.

[0166] Network interface 624 may include a SoC that provides modulation and demodulation functions and enables one or more controllers 636 to communicate over a wireless network. Network interface 624 may include an RF front-end for up-conversion from baseband to RF and down-conversion from RF to baseband. Frequency conversion can be performed using well-known processes and / or using superheterodyne processes. In some examples, the RF front-end functionality may be provided by a separate chip. The network interface may include wireless functions for communication via LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0167] The vehicle 600 may further include one or more data storage units 628, which may be off-chip (e.g., off-SoC). The data storage unit 628 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash memory, hard disk, and / or other components and / or devices capable of storing at least one bit of data.

[0168] The vehicle 600 may also include one or more GNSS sensors 658. One or more GNSS sensors 658 (e.g., GPS, assisted GPS sensors, differential GPS (DGPS) sensors, etc.) are used to assist in mapping, sensing, occupancy grid generation, and / or path planning functions. Any number of GNSS sensors 658 can be used, including, for example, but not limited to, GPS with a USB connector having an Ethernet-to-serial (RS-232) bridge.

[0169] Vehicle 600 may also include one or more radar sensors 660. Even in dark and / or inclement weather conditions, vehicle 600 can use one or more radar sensors 660 for remote vehicle detection. The radar functional safety level can be ASIL B. One or more radar sensors 660 can use CAN and / or bus 602 (e.g., to transmit data generated by one or more radar sensors 660) for control and access to target tracking data; in some examples, raw data is accessed via Ethernet. Various radar sensor types can be used. For example, but not limited to, one or more radar sensors 660 can be used for front, rear, and side radar applications. In some examples, pulse-Doppler radar sensors are used.

[0170] One or more radar sensors 660 may include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range side coverage, etc. In some examples, long-range radar can be used for adaptive cruise control functions. Long-range radar systems can provide a wide field of view, for example, within a 250-meter range, achieved through two or more independent scans. One or more radar sensors 660 can help distinguish between static and moving objects and can be used by ADAS systems for emergency braking assistance and forward collision warning. Long-range radar sensors may include monostatic multimode radars with multiple (e.g., six or more) fixed radar antennas and high-speed CAN and FlexRay interfaces. In an example with six antennas, the four central antennas can create a focused beam pattern designed to record 600 elements of the environment around the vehicle at high speed with minimal traffic interference from adjacent lanes. The other two antennas can expand the field of view, enabling rapid detection of vehicles entering or leaving the vehicle within 600 lanes.

[0171] For example, a mid-range radar system may include a range of up to 660 meters (front) or 80 meters (rear), and a field of view of up to 42 degrees (front) or 650 degrees (rear). Short-range radar systems may include, but are not limited to, radar sensors designed to be mounted at both ends of the rear bumper. When mounted at both ends of the rear bumper, such radar sensor systems may generate two beams, continuously monitoring the blind spots behind and beside the vehicle.

[0172] ADAS systems can use short-range radar systems for blind spot detection and / or lane change assistance.

[0173] Vehicle 600 may also include one or more ultrasonic sensors 662. One or more ultrasonic sensors 662 may be located at the front, rear, and / or sides of vehicle 600 and may be used for parking assistance and / or creating and updating occupancy grids. Various ultrasonic sensors 662 may be used, and different ultrasonic sensors 662 may be used for different detection ranges (e.g., 2.5m, 4m). One or more ultrasonic sensors 662 may operate at the ASIL B functional safety level.

[0174] Vehicle 600 may include one or more lidar sensors 664. The one or more lidar sensors 664 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The functional safety level of the one or more lidar sensors 664 may be ASIL B. In some examples, vehicle 600 may include multiple lidar sensors 664 (e.g., two, four, six, etc.) that can use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0175] In some examples, one or more LiDAR sensors 664 can provide a 360-degree field of view of objects and a list of their distances. One or more commercially available LiDAR sensors 664 have an advertised range of approximately 600m, an accuracy of 2cm-3cm, and support, for example, a 600Mbps Ethernet connection. In some examples, one or more non-protruding LiDAR sensors 664 can be used. In the examples of this type, one or more LiDAR sensors 664 can be implemented as a small device that can be embedded in the front, rear, sides, and / or corners of a vehicle 600. In the examples of this type, one or more LiDAR sensors 664 can provide a horizontal field of view of up to 120 degrees and a vertical field of view of 35 degrees, with a range of 200m even for low-reflectivity objects. A front-mounted one or more LiDAR sensors 664 can be configured with a horizontal field of view between 45 and 135 degrees. Figure 6B Example long-range and short-range horizontal fields of view of the LiDAR sensor 664 with an example mounting position above the windshield are shown, but other configurations are also possible, such as including a grille-mounted LiDAR sensor 664 (e.g., as...). Figure 6A Configuration of (as shown) and / or a roof-mounted LiDAR scanner (e.g., for a data acquisition vehicle).

[0176] In some examples, lidar technology, such as 3D flash lidar, can also be used. 3D flash lidar uses a laser flash as a transmission source, illuminating approximately 200 meters around the vehicle. A flash lidar device includes a receiver that records the laser pulse transmission time and reflected light at each pixel, with each pixel corresponding to the range from the vehicle to the object. Flash lidar can utilize each laser flash to generate a highly accurate, distortion-free environmental image. In some examples, four flash lidar sensors can be deployed, one on each side of the vehicle 600. Available 3D flash lidar systems include solid-state 3D staring array lidar cameras with no moving parts other than a fan (e.g., a non-scanning lidar device). The flash lidar device can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture reflected laser light in the form of a 3D distance point cloud and co-registered intensity data. By using flash lidar, and because flash lidar is a solid-state device with no moving parts, one or more lidar sensors 664 can be less susceptible to motion blur, vibration, and / or shock.

[0177] The vehicle may also include one or more IMU sensors 666. In some examples, one or more IMU sensors 666 may be located at the center of the rear axle of the vehicle 600. One or more IMU sensors 666 may include, for example, but not limited to, one or more accelerometers, one or more magnetometers, one or more gyroscopes, one or more magnetic compasses, and / or other sensor types. In some examples, such as in a six-axis application, one or more IMU sensors 666 may include accelerometers and gyroscopes, while in a nine-axis application, one or more IMU sensors 666 may include accelerometers, gyroscopes, and magnetometers.

[0178] In some embodiments, one or more IMU sensors 666 can be implemented as a miniaturized, high-performance GPS-assisted inertial navigation system (GPS / INS) that combines a microelectromechanical system (MEMS) inertial sensor, a high-sensitivity GPS receiver, and an advanced Kalman filtering algorithm to provide estimates of position, velocity, and attitude. Thus, in some examples, one or more IMU sensors 666 can enable vehicle 600 to estimate heading by directly observing and correlating velocity changes from GPS to one or more IMU sensors 666, without requiring input from magnetic sensors. In some examples, one or more IMU sensors 666 and one or more GNSS sensors 658 can be combined in a single integrated unit.

[0179] The vehicle may include one or more microphones 696 placed inside and / or around the vehicle 600. One or more microphones 696 may be used for emergency vehicle detection and identification, etc.

[0180] The vehicle may also include any number of camera types, including one or more stereo cameras 668, one or more wide-angle cameras 670, one or more infrared cameras 672, one or more surround cameras 674, one or more long-range and / or mid-range cameras 698, and / or other camera types. The cameras can be used to capture image data of the entire perimeter of the vehicle 600. The types of cameras used depend on the embodiment and requirements of the vehicle 600, and any combination of camera types can be used to provide the necessary coverage around the vehicle 600. Furthermore, the number of cameras can vary depending on the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or other numbers of cameras. As an example, the cameras may support, but are not limited to, Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet. This document will refer to... Figure 6A and Figure 6B Describe each camera in more detail.

[0181] Vehicle 600 may also include one or more vibration sensors 642. One or more vibration sensors 642 can measure vibrations of vehicle components, such as axles. For example, changes in vibration may indicate changes in road surface. In another example, when two or more vibration sensors 642 are used, differences between vibrations can be used to determine friction or slippage on the road surface (e.g., when the vibration difference is between a driven shaft and a freely rotating shaft).

[0182] Vehicle 600 may include ADAS system 638. In some examples, ADAS system 638 may include SoC. ADAS system 638 may include automatic / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keeping assist (LKA), blind spot warning (BSW), rear cross traffic warning (RCTW), collision warning system (CWS), lane centering (LC) and / or other features and functions.

[0183] The ACC system may use one or more radar sensors 660, one or more lidar sensors 664, and / or one or more cameras. The ACC system may include longitudinal ACC and / or lateral ACC. Longitudinal ACC monitors and controls the distance to the vehicle directly in front of vehicle 600 and automatically adjusts the vehicle speed to maintain a safe distance. Lateral ACC performs distance keeping and suggests lane changes to vehicle 600 if necessary. Lateral ACC is associated with other ADAS applications such as LCA and CWS.

[0184] CACC uses information from other vehicles, which can be received indirectly from other vehicles via a wireless link or a network connection (e.g., via the Internet) through network interface 624 and / or one or more wireless antennas 626. The direct link can be provided by a vehicle-to-vehicle (V2V) communication link, while the indirect link can be an infrastructure-to-vehicle (I2V) communication link. Generally, the V2V communication concept provides information about vehicles ahead (e.g., vehicles directly in front of vehicle 600 and in the same lane), while the I2V communication concept provides information about traffic ahead. A CACC system may include one or both I2V and V2V information sources. By taking into account information about vehicles ahead of vehicle 600, CACC may be more reliable and has the potential to improve traffic flow smoothness and reduce congestion on the road.

[0185] The Forward-Facing Warning (FCW) system is designed to alert the driver to hazards so that the driver can take corrective action. The FCW system uses a forward-facing camera and / or one or more radar sensors 660 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which are electrically coupled to driver feedback, such as a display, speaker, and / or vibration components. The FCW system can provide warnings such as audible, visual, haptic, and / or rapid braking pulses.

[0186] The AEB system detects an impending forward collision with another vehicle or other object. If the driver does not take corrective action within a specified time or distance parameter, the AEB system may automatically apply the brakes. The AEB system may use one or more front-facing cameras and / or one or more radar sensors 660 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. When the AEB system detects a hazard, it typically first alerts the driver to take corrective action to avoid a collision. If the driver does not take corrective action, the AEB system may automatically apply the brakes to prevent or at least mitigate the effects of the anticipated collision. The AEB system may include technologies such as dynamic brake support and / or collision proximity braking.

[0187] The Lane Departure Warning (LDW) system provides visual, auditory, and / or tactile warnings, such as steering wheel or seat vibrations, to alert the driver when the vehicle crosses lane markings. The LDW system does not activate when the driver indicates intentional lane departure by activating a turn signal. The LDW system may use a front-facing camera coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to actuator feedback, such as a display, speaker, and / or vibration assembly.

[0188] The LKA system is a variant of the LDW system. If the vehicle begins to leave the lane at 60°, the LKA system provides steering input or braking to correct the vehicle's 60° deviation.

[0189] The BSW system detects and warns drivers of vehicles within the blind spot. The BSW system can provide visual, auditory, and / or tactile alerts to indicate unsafe merging or lane changing. Additional warnings may be provided when the driver uses turn signals. The BSW system may utilize a rear-facing camera and / or radar sensor 660, coupled to a dedicated processor, DSP, FPGA, and / or ASIC, and electrically coupled to driver feedback, such as a display, speaker, and / or vibration assembly.

[0190] When the vehicle 600 is reversing and detects an object outside the range of the rear camera, the RCTW system can provide visual, auditory, and / or tactile notifications. Some RCTW systems include AEB (Autonomous Emergency Braking) to ensure the application of the vehicle's brakes to avoid a collision. The RCTW system may use one or more rear-facing radar sensors 660, which are coupled to a dedicated processor, DSP, FPGA, and / or ASIC, electrically coupled to actuator feedback, such as a display, speaker, and / or vibration assembly.

[0191] Traditional ADAS systems can be prone to false positives, which can be frustrating and distracting for drivers, but usually do not lead to catastrophic consequences because the ADAS system alerts the driver and allows them to determine whether a safe situation truly exists and take appropriate action. However, in an autonomous vehicle 600, in the event of conflicting results, the vehicle 600 itself must decide whether to listen to the results from the main computer or the auxiliary computer (e.g., the first controller 636 or the second controller 636). For example, in some embodiments, the ADAS system 638 may be a backup and / or auxiliary computer for providing perception information to a backup computer module. A backup computer rationality monitor may run redundant different software on hardware components to detect faults in perception and dynamic driving tasks. The output from the ADAS system 638 may be provided to a monitoring MCU. If the outputs of the main computer and the auxiliary computer conflict, the monitoring MCU must determine how to reconcile the conflict to ensure safe operation.

[0192] In some examples, the master computer can be configured to provide a confidence score to the monitoring MCU, indicating the master computer's confidence level in a selected result. If the confidence score exceeds a threshold, the monitoring MCU can follow the master computer's direction regardless of whether the auxiliary computer provides conflicting or inconsistent results. If the confidence score does not meet the threshold, and the master computer and the auxiliary computer indicate different results (e.g., conflict), the monitoring MCU can arbitrate between the computers to determine the appropriate result.

[0193] The monitoring MCU can be configured to run one or more trained and configured neural networks to determine the conditions under which the auxiliary computer provides a false alarm based on the outputs of the main computer and the auxiliary computer. Thus, one or more neural networks in the monitoring MCU can learn when the output of the auxiliary computer is reliable and when it is not. For example, when the auxiliary computer is a radar-based FCW system, one or more neural networks in the monitoring MCU can learn when the FCW system identifies a metallic object that is not actually dangerous, such as a drain grille or manhole cover that triggers an alarm. Similarly, when the auxiliary computer is a camera-based LDW system, the neural network in the monitoring MCU can learn to override the LDW when a bicycle or pedestrian is present and lane departure is actually the safest maneuver. In embodiments that include one or more neural networks running on the monitoring MCU, the monitoring MCU may include at least one of a DLA or GPU adapted to run one or more neural networks with associated memory. In a preferred embodiment, the monitoring MCU may include and / or include components that are SoC 604.

[0194] In other examples, ADAS system 638 may include an auxiliary computer that performs ADAS functions using conventional computer vision rules. Therefore, the auxiliary computer can use classic computer vision rules (if-then), and the presence of one or more neural networks in the monitoring MCU can improve reliability, safety, and performance. For example, diverse implementations and intentional non-identification make the entire system more fault-tolerant, especially to failures caused by software (or hardware / software interface) functionality. For instance, if a software defect or error exists in the software running on the main computer, and different software code running on the auxiliary computer provides the same overall result, the monitoring MCU can have greater confidence in the correctness of the overall result, and the software or hardware defect on the main computer did not lead to a major error.

[0195] In some examples, the output of ADAS system 638 may be fed into the perception block and / or the dynamic drive task block of the host computer. For example, if ADAS system 638 indicates a forward collision warning due to an object directly in front, the perception block may use this information when identifying the object. In other examples, as described herein, the secondary computer may have its own trained neural network, thereby reducing the risk of false positives.

[0196] Vehicle 600 may also include an infotainment SoC 630 (e.g., an in-vehicle infotainment system (IVI)). Although shown and described as an SoC, the infotainment system may not be an SoC and may include two or more discrete components. The infotainment SoC 630 may include a combination of hardware and software that can be used to provide audio (e.g., music, personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., television, movies, streaming media, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and / or provide information services to vehicle 600 (e.g., navigation system, rear parking assist, radio data system, vehicle-related information such as fuel level, total distance traveled, brake fuel level, engine oil level, door opening / closing, air filter information, etc.). For example, the infotainment SoC 630 may include a radio, disk player, navigation system, video player, USB and Bluetooth connectivity, in-vehicle computer, in-vehicle entertainment, Wi-Fi, steering wheel audio controls, hands-free voice control, head-up display (HUD), HMI display 634, telecom device, control panel (e.g., for controlling and / or interacting with various components, features, and / or systems), and / or other components. The infotainment SoC 630 may also be used to provide information to vehicle users (e.g., visual and / or auditory), such as information from ADAS system 638, autonomous driving information such as planned vehicle maneuvers, trajectory, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0197] The infotainment SoC 630 may include GPU functionality. The infotainment SoC 630 can communicate with other devices, systems, and / or components of the vehicle 600 via bus 602 (e.g., CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 630 may be coupled to a monitoring MCU, allowing the GPU of the infotainment system to perform self-driving functions in the event of a failure of one or more main controllers 636 (e.g., the main computer and / or backup computer of the vehicle 600). In such an example, the infotainment SoC 630 may place the vehicle 600 into a driver-to-safe parking mode, as described herein.

[0198] Vehicle 600 may also include an instrument cluster 632 (e.g., a digital instrument panel, electronic instrument cluster, etc.). The instrument cluster 632 may include a controller and / or a supercomputer (e.g., a discrete controller or supercomputer). The instrument cluster 632 may include a set of instruments such as a speedometer, fuel level, oil pressure, tachometer, odometer, turn signals, shift position indicator, one or more seatbelt warning lights, one or more parking brake warning lights, one or more engine malfunction lights, airbag (SRS) system information, lighting controls, safety system controls, navigation information, etc. In some examples, information may be displayed and / or shared between the infotainment SoC 630 and the instrument cluster 632. Accordingly, the instrument cluster 632 may be included as part of the infotainment SoC 630, and vice versa.

[0199] Figure 6D It is one or more cloud-based servers according to some embodiments of this disclosure and Figure 6A A system diagram illustrating communication between exemplary autonomous vehicles 600 is provided. System 676 may include one or more servers 678, one or more networks 690, and vehicles, including vehicle 600. One or more servers 678 may include multiple GPUs 684(A)-684(H) (collectively referred to as GPU 684), PCIe switches 682(A)-682(D) (collectively referred to as PCIe switch 682), and / or CPUs 680(A)-680(B) (collectively referred to herein as CPU 680). GPUs 684, CPUs 680, and PCIe switches may be interconnected via high-speed interconnects, such as, but not limited to, NVIDIA-developed NVLink interface 688 and / or PCIe connection 686. In some examples, GPUs 684 are connected via NVLink and / or NV switch SoCs, and GPUs 684 and PCIe switches 682 are connected via PCIe interconnects. Although eight GPUs 684, two CPUs 680, and two PCIe switches are illustrated, this is not intended to be limiting. According to an embodiment, each of one or more servers 678 may include any number of GPUs 684, CPUs 680, and / or PCIe switches. For example, one or more servers 678 may each include eight, sixteen, thirty-two, and / or more GPUs 684.

[0200] One or more servers 678 may receive image data representing images from vehicles via one or more networks 690, showing unexpected or changed road conditions, such as recently started roadwork. One or more servers 678 may send neural network 692, updated neural network 692, and / or map information 694, including information about traffic and road conditions, to vehicles via one or more networks 690. Updates to map information 694 may include updates to HD map 622, such as information about construction sites, potholes, detours, floods, and / or other obstacles. In some examples, neural network 692, updated neural network 692, and / or map information 694 may originate from new training and / or experience represented in data received from any number of vehicles in the environment, and / or based on training performed in a data center (e.g., using one or more servers 678 and / or other servers).

[0201] One or more servers 678 can be used to train machine learning models (e.g., neural networks) based on training data. Training data may be generated using vehicles and / or generated in simulations (e.g., using a game engine). In some examples, the training data is labeled (e.g., the neural network benefits from supervised learning) and / or undergoes other preprocessing, while in other examples, the training data is unlabeled and / or preprocessed (e.g., the neural network does not require supervised learning). Training can be performed according to any one or more classes of machine learning techniques, including but not limited to: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, joint learning, transfer learning, feature learning (including principal component analysis and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including alternative dictionary learning), rule-based machine learning, anomaly detection, and any variations or combinations thereof. Once the machine learning model is trained, it can be used by vehicles (e.g., transmitted to vehicle 690 via one or more networks, and / or the machine learning model can be used by one or more servers 678 for remote vehicle monitoring).

[0202] In some examples, one or more servers 678 may receive data from a vehicle and apply the data to state-of-the-art real-time neural networks for real-time intelligent inference. One or more servers 678 may include a deep learning supercomputer and / or a dedicated AI computer powered by a GPU 684, such as the DGX and DGX Station machines developed by NVIDIA. However, in some examples, one or more servers 678 may include a deep learning infrastructure in a data center using only CPU power.

[0203] The deep learning infrastructure of one or more servers 678 can perform rapid real-time inference and use this capability to assess and verify the health status of the processors, software, and / or associated hardware in vehicle 600. For example, the deep learning infrastructure can receive periodic updates from vehicle 600, such as sequences of images and / or objects located by vehicle 600 in the image sequence (e.g., through computer vision and / or other machine learning object classification techniques). The deep learning infrastructure can run its own neural network to identify objects and compare them with objects identified by vehicle 600. If the results do not match and the infrastructure concludes that the AI ​​in vehicle 600 has malfunctioned, one or more servers 678 can send a signal to vehicle 600 instructing the fail-safe computer of vehicle 600 to take control, notify passengers, and complete a safe stopping operation.

[0204] For inference, one or more servers 678 may include one or more GPUs 684 and one or more programmable inference accelerators (such as NVIDIA's TensorRT). The combination of GPU-powered servers and inference acceleration can enable real-time responses. In other examples, such as where performance is less critical, inference can be performed using servers powered by CPUs, FPGAs, and other processors.

[0205] Reasoning and training logic

[0206] One or more embodiments may be implemented using inference and / or training logic for performing inference and / or training operations. Details regarding the inference and / or training logic are provided below.

[0207] In at least one embodiment, the inference and / or training logic may include, but is not limited to, code and / or data storage for storing forward and / or output weights and / or input / output data, and / or other parameters for configuring neurons or layers of a neural network trained and / or used for inference in one or more embodiments. In at least one embodiment, the training logic may include or be coupled to code and / or data storage for storing graph code or other software to control timing and / or sequence, wherein weight and / or other parameter information is loaded to configure logic including integer and / or floating-point units (collectively, an arithmetic logic unit (ALU)). In at least one embodiment, code (such as graph code) loads weight or other parameter information into the processor ALU based on the architecture of the neural network to which the code corresponds. In at least one embodiment, the code and / or data storage is stored during training and / or inference using one or more embodiments, incorporating weight parameters and / or input / output data of each layer of a neural network trained or used in one or more embodiments during forward propagation of input / output data and / or weight parameters. In at least one embodiment, any portion of the code and / or data storage may be included within other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory.

[0208] In at least one embodiment, any portion of the code and / or data storage may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data storage may be cache memory, dynamic random access memory (“DRAM”), static random access memory (“SRAM”), non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the choice of whether the code and / or data storage is internal or external to the processor, for example, or including DRAM, SRAM, flash memory, or some other storage type, may depend on the available on-chip versus off-chip storage, the latency requirements of the training and / or inference functions being performed, the batch size of the data used in the inference and / or training of the neural network, or some combination of these factors.

[0209] In at least one embodiment, the inference and / or training logic may include, but is not limited to, code and / or data storage for storing backpropagation and / or output weights and / or input / output data corresponding to neurons or layers of a neural network trained and / or used for inference in one or more embodiments. In at least one embodiment, during training and / or inference using one or more embodiments, the code and / or data storage is stored in conjunction with the weight parameters and / or input / output data of each layer of the neural network trained or used in one or more embodiments during backpropagation of the input / output data and / or weight parameters. In at least one embodiment, the inference and / or training logic may include or be coupled to code and / or data storage for storing graph code or other software to control timing and / or sequence, wherein weight and / or other parameter information is loaded to configure logic including integer and / or floating-point units (collectively, an arithmetic logic unit (ALU)). In at least one embodiment, code (such as graph code) loads weight or other parameter information into the processor ALU based on the architecture of the neural network to which the code corresponds. In at least one embodiment, any portion of the code and / or data storage may be included together with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory. In at least one embodiment, any portion of the code and / or data storage may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data storage may be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the choice of whether the code and / or data storage is internal or external to the processor, for example, including DRAM, SRAM, flash memory, or some other type of storage, may depend on the available on-chip versus off-chip storage, the latency requirements of the training and / or inference functions being performed, the batch size of the data used in the inference and / or training of the neural network, or some combination of these factors.

[0210] In at least one embodiment, the code and / or data storage, and the code and / or data storage itself, may be separate storage structures. In at least one embodiment, the code and / or data storage, and the code and / or data storage itself, may be the same storage structure. In at least one embodiment, the code and / or data storage, and the code and / or data storage itself, may be partially combined and partially separated. In at least one embodiment, the code and / or data storage, and any portion thereof, may be included together with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory.

[0211] In at least one embodiment, the inference and / or training logic may include, but is not limited to, one or more arithmetic logic units (“ALUs”) (including integer and / or floating-point units) for performing logical and / or mathematical operations at least in part based on or instructed by training and / or inference code (e.g., graph code), the results of which may produce activations stored in activation storage (e.g., output values ​​from layers or neurons within a neural network), which are functions of input / output and / or weight parameter data stored in code and / or data storage. In at least one embodiment, activations stored in activation storage are generated based on linear algebra and / or matrix-based mathematics performed by the ALU in response to execution instructions or other code, wherein weight values ​​stored in and / or data storage serve as operands, and other values, such as bias values, gradient information, momentum values, or other parameters or hyperparameters, may be stored in code and / or data storage or other on-chip or off-chip storage.

[0212] In at least one embodiment, one or more processors or other hardware logic devices or circuits include one or more ALUs, while in another embodiment, one or more ALUs may be external to the processor or other hardware logic device or the circuitry using them (e.g., a coprocessor). In at least one embodiment, ALUs may be included within an execution unit of a processor, or otherwise included in an ALU bank accessible by the execution unit of the processor, which may be within the same processor or distributed among different processors of different types (e.g., a central processing unit, a graphics processing unit, a fixed-function unit, etc.). In at least one embodiment, code and / or data storage, code and / or data storage, and activation storage may share the processor or other hardware logic device or circuitry, while in another embodiment, they may be in different processors or other hardware logic devices or circuitry, or some combination of the same and different processors or other hardware logic devices or circuitry. In at least one embodiment, any portion of the activation storage may be included together with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory. Furthermore, inference and / or training code may be stored together with other code accessible to the processor or other hardware logic or circuitry, and may be retrieved and / or processed using the processor’s fetch, decode, schedule, execute, exit, and / or other logic circuitry.

[0213] In at least one embodiment, the active memory may be a cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other memory. In at least one embodiment, the active memory may be entirely or partially located within or outside one or more processors or other logic circuits. In at least one embodiment, the choice of whether the active memory is internal or external to the processor, for example, or including DRAM, SRAM, flash memory, or certain other memory types, may depend on the availability of on-chip versus off-chip memory, latency requirements for performing training and / or inference functions, batch size of data used in inference and / or training the neural network, or some combination of these factors. In at least one embodiment, the inference and / or training logic may be used in conjunction with an application-specific integrated circuit (“ASIC”), such as those from Google. Processing unit, from Graphcore TM The inference processing unit (IPU) or from Intel. (e.g., "Lake Crest") processor. In at least one embodiment, the inference and / or training logic may be used in conjunction with central processing unit ("CPU") hardware, graphics processing unit ("GPU") hardware, or other hardware (e.g., field-programmable gate array ("FPGA")).

[0214] In at least one embodiment, the inference and / or training logic may include, but is not limited to, hardware logic, wherein computational resources, along with weight values ​​or other information corresponding to one or more layers of neurons within a neural network, are used dedicatedly or otherwise exclusively. In at least one embodiment, the inference and / or training logic may be used in conjunction with an application-specific integrated circuit (ASIC), such as those from Google. Processing unit, from Graphcore TM The inference processing unit (IPU) or from Intel. (e.g., "Lake Crest") processor. In at least one embodiment, the inference and / or training logic may be used in conjunction with central processing unit (CPU) hardware, graphics processing unit (GPU) hardware, or other hardware (e.g., field-programmable gate array (FPGA)). In at least one embodiment, the inference and / or training logic includes, but is not limited to, code and / or data storage and code and / or data storage, which may be used to store code (e.g., graph code), weight values, and / or other information, including bias values, gradient information, momentum values, and / or other parameter or hyperparameter information. In at least one embodiment, each of the code and / or data storage and code and / or data storage is associated with a dedicated computing resource (e.g., computing hardware and computing hardware). In at least one embodiment, each of the computing hardware and computing hardware includes one or more ALUs that perform mathematical functions (e.g., linear algebraic functions) only on the information stored in the code and / or data storage and code and / or data storage, and the results are stored in active memory.

[0215] In at least one embodiment, each of the code and / or data storage and the corresponding computing hardware corresponds to a different layer of the neural network, such that an activation obtained from one storage / computation pair of the code and / or data storage and computing hardware is provided as input to the next storage / computation pair of the code and / or data storage and computing hardware to reflect the conceptual organization of the neural network. In at least one embodiment, each storage / computation pair may correspond to more than one neural network layer. In at least one embodiment, additional storage / computation pairs (not shown) may be included in the inference and / or training logic, either following or parallel to the storage / computation pair.

[0216] Example computing device

[0217] Figure 7This is a block diagram of an example computing device 700 suitable for implementing some embodiments of the present disclosure. The computing device 700 may include an interconnect system 702 directly or indirectly coupled to: a memory 704, one or more central processing units (CPUs) 706, one or more graphics processing units (GPUs) 708, a communication interface 710, input / output (I / O) ports 712, input / output components 714, a power supply 716, one or more presentation components 718 (e.g., displays), and one or more logic units 720. In at least one embodiment, one or more computing devices 700 may include one or more virtual machines (VMs), and / or any component thereof may include virtual components (e.g., virtual hardware components). For a non-limiting example, one or more GPUs 708 may include one or more vGPUs, one or more CPUs 706 may include one or more vCPUs, and / or one or more logic units 720 may include one or more virtual logic units. Therefore, one or more computing devices 700 may include discrete components (e.g., a complete GPU dedicated to computing device 700), virtual components (e.g., a portion of the GPU dedicated to computing device 700), or a combination thereof.

[0218] although Figure 7 The various modules are shown as being connected to lines via interconnect system 702, but this is not intended to be limiting, but merely for clarity. For example, in some embodiments, a presentation component 718 such as a display device can be considered as I / O component 714 (e.g., if the display is a touchscreen). As another example, CPU 706 and / or GPU 708 may include memory (e.g., memory 704 may also represent a storage device in addition to the memory of GPU 708, CPU 706, and / or other components). Accordingly, Figure 7 The computing devices mentioned are merely illustrative. No distinction is made between "workstation," "server," "laptop," "desktop," "tablet," "client device," "mobile device," "handheld device," "game console," "electronic control unit (ECU)," "virtual reality system," and / or other device or system types, as is the case in [the context of the previous sentence]. Figure 7 As envisioned within the scope of computing devices.

[0219] Interconnect system 702 may represent one or more links or buses, such as address buses, data buses, control buses, or combinations thereof. Interconnect system 702 may include one or more bus or link types, such as Industry Standard Architecture (ISA) buses, Extended Industry Standard Architecture (EISA) buses, Video Electronics Standards Association (VESA) buses, Peripheral Component Interconnect (PCI) buses, Peripheral Component Interconnect Through (PCIE) buses, and / or other types of buses or links. In some embodiments, there is a direct connection between components. For example, CPU 706 may be directly connected to memory 704. Furthermore, CPU 706 may be directly connected to GPU 708. In cases where there is a direct connection or point-to-point connection between components, interconnect system 702 may include a PCIe link for performing the connection. In these examples, a PCI bus is not required in computing device 700.

[0220] The memory 704 may include any of a variety of computer-readable media. Computer-readable media can be any available medium accessible by the computing device 700. Computer-readable media may include volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, computer-readable media may include computer storage media and communication media.

[0221] Computer storage media may include volatile and non-volatile media and / or removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 704 may store computer-readable instructions (e.g., instructions representing one or more programs and / or one or more program elements), such as an operating system. Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other storage technologies, CD-ROM, digital versatile disk (DVD) or other optical disc storage, cassette tape, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by computing device 700. As used herein, computer storage media itself does not include signals.

[0222] Computer storage media can contain computer-readable instructions, data structures, program modules, and / or other data types contained in modulated data signals (such as carrier waves or other transmission mechanisms), and include any information delivery medium. The term "modulated data signal" can refer to a signal whose one or more characteristics are set or altered in a manner that encodes information in the signal. By way of example and not limitation, computer storage media can include wired media, such as wired networks or direct wired connections, and wireless media, such as acoustic, RF, infrared, and other wireless media. Any combination of the foregoing should also be included within the scope of computer-readable media.

[0223] One or more CPUs 706 may be configured to execute at least some computer-readable instructions to control one or more components of computing device 700 to perform one or more methods and / or processes described herein. Each of the one or more CPUs 706 may include one or more cores capable of processing multiple software threads simultaneously (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.). The one or more CPUs 706 may include any type of processor and may include different types of processors depending on the type of computing device 700 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 700, the processor may be an advanced RISC machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplementary coprocessors (e.g., math coprocessors), computing device 700 may also include one or more CPUs 706.

[0224] In addition to, or selected from, one or more CPUs 706, one or more GPUs 708 may be configured to execute at least some computer-readable instructions to control one or more components of computing device 700 to perform one or more methods and / or processes described herein. One or more GPUs 708 may be integrated GPUs (e.g., having one or more CPUs 706 and / or one or more GPUs 708 may be discrete GPUs). In embodiments, one or more GPUs 708 may be coprocessors of one or more CPUs 706. Computing device 700 may use one or more GPUs 708 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, one or more GPUs 708 may be used for general-purpose computing on a GPU (GPGPU). One or more GPUs 708 may include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. One or more GPUs 708 may generate pixel data for an output image in response to rendering commands (e.g., rendering commands received via a host interface from one or more CPUs 706). One or more GPUs 708 may include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. Display memory may be included as part of memory 704. One or more GPUs 708 may include two or more GPUs operating in parallel (e.g., via a link). The link may directly connect the GPUs (e.g., using NVLINK) or connect the GPUs via a switch (e.g., using NVSwitch). When combined, each GPU 708 may generate pixel data or GPGPU data for different portions of the output or different outputs (e.g., a first GPU for a first image, a second GPU for a second image). Each GPU may include its own memory or may share memory with other GPUs.

[0225] In addition to one or more CPUs 706 and / or one or more GPUs 708, or selected from one or more CPUs 706 and / or one or more GPUs 708, one or more logic units 720 may be configured to execute at least some computer-readable instructions to control one or more components of computing device 700 to perform one or more methods and / or processes described herein. In embodiments, one or more CPUs 706, one or more GPUs 708, and / or one or more logic units 720 may execute any combination of methods, processes, and / or portions thereof discretely or jointly. One or more logic units 720 may be part of and / or integrated into one or more CPUs 706 and / or GPUs 708, and / or one or more logic units 720 may be discrete components or otherwise located external to one or more CPUs 706 and / or one or more GPUs 708. One or more logic units 720 may be coprocessors of one or more CPUs 706 and / or one or more GPUs 708.

[0226] Examples of one or more logic units 720 include one or more processing cores and / or components thereof, such as a data processing unit (DPU), a tensor core (TC), a tensor processing unit (TPU), a pixel vision core (PVC), a vision processing unit (VPU), a graphics processing cluster (GPC), a texture processing cluster (TPC), a streaming multiprocessor (SM), a tree traversal unit (TTU), an artificial intelligence accelerator (AIA), a deep learning accelerator (DLA), an arithmetic logic unit (ALU), an application-specific integrated circuit (ASIC), a floating-point unit (FPU), input / output (I / O) elements, peripheral component interconnect (PCI) or peripheral component interconnect pass-through (PCIe) elements, and / or the like.

[0227] The communication interface 710 may include one or more receivers, transmitters, and / or transceivers that allow the computing device 700 to communicate with other computing devices via an electronic communication network including wired and / or wireless communications. The communication interface 710 may include components and functions to allow communication over any of a variety of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication over Ethernet or InfiniBand), low-power wide-area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, one or more logic units 720 and / or the communication interface 710 may include one or more data processing units (DPUs) for directly transmitting data received via a network and / or via interconnect system 702 to one or more GPUs 708 (e.g., their memory).

[0228] I / O port 712 allows computing device 700 to be logically coupled to other devices, including I / O component 714, one or more presentation components 718, and / or other components, some of which may be built-in (e.g., integrated) within computing device 700. Illustrative I / O component 714 includes microphones, mice, keyboards, joysticks, game pads, game controllers, satellite antennas, scanners, printers, wireless devices, etc. I / O component 714 can provide a natural user interface (NUI) that processes user-generated air gestures, voice, or other physiological input. In some cases, input can be transmitted to appropriate network elements for further processing. NUI can implement any combination of voice recognition, stylus recognition, facial recognition, biometrics, on-screen and near-screen gesture recognition, air gestures, head and eye tracking, and touch recognition associated with the display of computing device 700 (described in more detail below). Computing device 700 may include depth cameras, such as stereo camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof, for gesture detection and recognition. In addition, computing device 700 may include an accelerometer or gyroscope that allows motion detection (e.g., as part of an inertial measurement unit (IMU)). In some examples, the output of the accelerometer or gyroscope may be used by computing device 700 to render immersive augmented reality or virtual reality.

[0229] Power supply 716 may include hard-wired power supply, battery power supply, or a combination thereof. Power supply 716 may supply power to computing device 700 to allow the components of computing device 700 to operate.

[0230] One or more presentation components 718 may include displays (e.g., monitors, touchscreens, television screens, head-up displays (HUDs), other display types, or combinations thereof), speakers, and / or other presentation components. One or more presentation components 718 may receive data from other components (e.g., one or more GPUs 708, one or more CPUs 706, DPUs, etc.) and output data (e.g., as images, videos, sounds, etc.).

[0231] Example Data Center

[0232] Figure 8 An example data center 800 that can be used in at least one embodiment of this disclosure is shown. The data center 800 may include a data center infrastructure layer 810, a framework layer 820, a software layer 830, and / or an application layer 840.

[0233] like Figure 8 As shown, the data center infrastructure layer 810 may include a resource coordinator 812, packet computing resources 814, and node computing resources (“nodes CRs”) 816(1)-816(N), where “N” represents any integer. In at least one embodiment, nodes CRs 816(1)-816(N) may include, but are not limited to, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field-programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), storage devices (e.g., dynamic read-only memory), and in some embodiments, storage devices (e.g., solid-state or disk drives), network input / output (NW I / O) devices, network switches, virtual machines (VMs), power modules and / or cooling modules, etc. One or more nodes CRs 816(1)-816(N) may correspond to a server having one or more of the aforementioned computing resources. In addition, in some embodiments, nodes CRs 816(1)-816(N) may include one or more virtual components, such as vGPU, vCPU and / or similar components, and / or one or more nodes CRs 816(1)-816(N) may correspond to virtual machines (VMs).

[0234] In at least one embodiment, the packet computing resource 814 may include individual packets of node CRs 816 located within one or more racks (not shown), or multiple racks located within data centers in different geographical locations (also not shown). Individual packets of node CRs 816 within the packet computing resource 814 may include packet computing, networking, memory, or storage resources, which may be configured or allocated to support one or more workloads. In at least one embodiment, multiple node CRs 816, including CPUs, GPUs, DPUs, and / or other processors, may be grouped within one or more racks to provide computing resources to support one or more workloads. One or more racks may also include any combination of any number of power modules, cooling modules, and / or network switches.

[0235] Resource coordinator 812 may be configured or otherwise control one or more nodes CRs816(1)-816(N) and / or grouped computing resources 814. In at least one embodiment, resource coordinator 812 may include a Software Design Infrastructure (SDI) management entity for data center 800. Resource coordinator 812 may include hardware, software, or some combination thereof.

[0236] In at least one embodiment, such as Figure 8 As shown, framework layer 820 may include job scheduler 833, configuration manager 834, resource manager 836, and / or distributed file system 838. Framework layer 820 may include frameworks for software 832 supporting software layer 830 and / or one or more applications 842 supporting application layer 840. Software 832 or one or more applications 842 may respectively include web-based service software or applications, such as services provided by Amazon Web Services, Google Cloud, and Microsoft Azure. Framework layer 820 may be, but is not limited to, a free and open-source software web application framework, such as Apache Spark™ (hereinafter referred to as "Spark"), which can utilize distributed file system 838 for large-scale data processing (e.g., "big data"). In at least one embodiment, job scheduler 833 may include Spark drivers to facilitate the scheduling of workloads supported by the various layers of data center 800. Configuration manager 834 may configure different layers, such as software layer 830 and framework layer 820, including Spark and distributed file system 838, to support large-scale data processing. Resource manager 836 may be able to manage cluster or group computing resources mapped to or allocated to support distributed file system 838 and job scheduler 833. In at least one embodiment, the cluster or group computing resources may include group computing resources 814 at data center infrastructure layer 810. Resource manager 836 may coordinate with resource coordinator 812 to manage these mapped or allocated computing resources.

[0237] In at least one embodiment, the software 832 included in the software layer 830 may include software used in at least a portion of the distributed file system 838 of the nodes CRs 816(1)-816(N), the grouped computing resources 814, and / or the framework layer 820. One or more types of software may include, but are not limited to, internet webpage search software, email virus scanning software, database software, and streaming video content software.

[0238] In at least one embodiment, the application 842 included in the application layer 840 may include one or more types of applications used by at least a portion of the nodes CRs 816(1)-816(N), the grouped computing resources 814 and / or the distributed file system 838 of the framework layer 820, but is not limited to any number of genomics applications, perceptual computing and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.) and / or other machine learning applications used in combination with one or more embodiments.

[0239] In at least one embodiment, any of the configuration manager 834, resource manager 836, and resource coordinator 812 can implement any number and type of self-modification actions based on any amount and type of data acquired in any technically feasible manner. Self-modification actions can protect data center operators of data center 800 from making potentially erroneous configuration decisions and may prevent underutilized and / or poorly performing portions of the data center.

[0240] According to one or more embodiments described herein, data center 800 may include tools, services, software, or other resources for training one or more machine learning models or using one or more machine learning models to predict or infer information. For example, one or more machine learning models may be trained by calculating weight parameters based on a neural network architecture using the software and / or computing resources described above with respect to data center 800. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks may be used to infer or predict information using the resources described above with respect to data center 800 by using weight parameters calculated through one or more training techniques (e.g., but not limited to the training techniques described herein).

[0241] In at least one embodiment, the data center 800 may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, and / or other hardware (or corresponding virtual computing resources) to perform training and / or inference using the resources described above. Furthermore, one or more of the software and / or hardware resources described above may be configured as services to allow a user to train or perform information inference, such as image recognition, speech recognition, or other artificial intelligence services.

[0242] Example network environment

[0243] A network environment suitable for implementing embodiments of this disclosure may include one or more client devices, servers, network attached storage (NAS), other backend devices, and / or other device types. Client devices, servers, and / or other device types (e.g., each device) may... Figure 7 Implemented on one or more instances of computing devices 700—for example, each device may include similar components, features, and / or functions of computing device 700. Additionally, backend devices (servers, NAS, etc.) may be included as part of data center 800, examples of which are referred to herein. Figure 8 To describe in more detail.

[0244] Components of a network environment can communicate with each other through one or more networks, whether wired, wireless, or a combination of both. This network can include multiple networks, or networks of networks. For example, the network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. In cases where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (along with other components) can provide wireless connectivity.

[0245] A compatible network environment may include one or more peer-to-peer network environments—in which case the server may not be included in the network environment—and one or more client-server network environments—in which case one or more servers may be included in the network environment. In a peer-to-peer network environment, the server functionality described herein can be implemented on any number of client devices.

[0246] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, and combinations thereof. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework supporting software at the software layer and / or application at the application layer. The software or one or more applications may respectively include web-based service software or applications. In embodiments, one or more client devices may use web-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a free and open-source software web application framework, for example, one that can use a distributed file system for large-scale data processing (e.g., "big data").

[0247] A cloud-based network environment can provide cloud computing and / or cloud storage for any combination of the computing and / or data storage functions (or one or more of them) described herein. Any of these various functions can be distributed from a central or core server (e.g., servers in one or more data centers) to multiple locations, which may be located in a state, region, country, globally, etc. If the connection to a user (e.g., a client device) is relatively close to one or more edge servers, one or more core servers may assign at least a portion of the functionality to one or more edge servers. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0248] One or more client devices may include the information described in this article. Figure 7 The client device may be, by way of example and not limitation, at least some of the components, features and functions of one or more example computing devices 700. As an example and not a limitation, the client device may be a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, camera, surveillance equipment or system, vehicle, ship, aircraft, virtual machine, drone, robot, handheld communication device, hospital equipment, gaming device or system, entertainment system, vehicle computer system, embedded system controller, remote control, device, consumer electronics device, workstation, edge device, any combination of these depicted devices or any other suitable device.

[0249] This disclosure can be described in the general context of computer code or machine-usable instructions, including computer-executable instructions, such as program modules, that are executed by a computer or other machine (such as a personal data assistant or other handheld device). Typically, program modules include routines, programs, objects, components, data structures, etc., and refer to code that performs a specific task or implements a specific abstract data type. This disclosure can be implemented in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, and more specialized computing devices. This disclosure can also be implemented in distributed computing environments, where tasks are performed by remote processing devices linked via a communication network.

[0250] Other variations are within the spirit of this disclosure. Therefore, although the disclosed technology is readily adaptable to various modifications and alternative constructions, certain embodiments thereof are illustrated in the accompanying drawings and have been described in detail above. However, it should be understood that the disclosure is not intended to be limited to one or more specific forms disclosed, but rather, it is intended to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of this disclosure as defined in the appended claims.

[0251] Unless otherwise stated or obviously contradicted by the context, the terms “a,” “an,” and “the,” and similar pronouns, used in the context of describing the disclosed embodiments (particularly in the context of the appended claims), should be interpreted as encompassing both singular and plural forms, rather than as definitions of the terms. Unless otherwise stated, the terms “comprising,” “having,” “including,” and “containing” should be interpreted as open-ended terms (meaning “including, but not limited to”). The term “connection” (wherein it is not modified, it refers to a physical connection) should be interpreted as partially or wholly included, attached to, or connected together, even with some intervening elements. Unless otherwise indicated herein, references to numerical ranges herein are intended only as a way of abbreviating each individual value falling within that range, and each individual value is incorporated into the specification as if it were separately described herein. Unless otherwise indicated or contradicted by the context, the use of the terms “set” (e.g., “item set”) or “subset” should be interpreted as a non-empty set comprising one or more members. Furthermore, unless otherwise indicated or contradicted by the context, the term “subset” of the corresponding set does not necessarily mean an appropriate subset of the corresponding set, but rather that the subset and the corresponding set can be equal.

[0252] Unless otherwise explicitly stated or clearly contradicted by the context, connective phrases such as “at least one of A, B, and C” or “at least one of A, B, and C” are understood in the context to generally refer to items, terms, etc., which can be A or B or C, or any non-empty subset of the set A, B, and C. For example, in an illustrative example of a set with three members, the connective phrases “at least one of A, B, and C” and “at least one of A, B, and C” refer to any of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Therefore, such connective language is generally not intended to imply that some embodiments require the presence of each of at least one of A, at least one of B, and at least one of C. Additionally, unless otherwise stated or contradicted by the context, the term “multiple” indicates a plural state (e.g., “multiple items” indicates multiple items). Multiple means at least two items, but more can be indicated if explicitly stated or by the context. Furthermore, unless otherwise stated or clearly understood from the context, the phrase “based on” means “at least partially based on” rather than “based on only”.

[0253] Unless otherwise indicated herein or clearly contradicted by the context, the operations of the processes described herein may be performed in any suitable order. In at least one embodiment, processes such as those described herein (or variations thereof and / or combinations thereof) are executed under the control of one or more computer systems configured with executable instructions and are implemented as code (e.g., executable instructions, one or more computer programs, or one or more application programs) executed jointly by hardware or a combination thereof on one or more processors. In at least one embodiment, the code is stored on a computer-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. In at least one embodiment, the computer-readable storage medium is a non-transitory computer-readable storage medium that excludes transient signals (e.g., propagating transient electrical or electromagnetic transmissions) but includes non-transitory data storage circuitry (e.g., buffers, caches, and queues) within a transceiver of transient signals. In at least one embodiment, code (e.g., executable code or source code) is stored on a set of one or more non-transitory computer-readable storage media (or other memory for storing executable instructions) on which executable instructions are stored, which, when executed by one or more processors of a computer system (i.e., as a result of execution), cause the computer system to perform the operations described herein. In at least one embodiment, the set of non-transitory computer-readable storage media comprises a plurality of non-transitory computer-readable storage media, and one or more of the various non-transitory storage media lack the complete code, but the plurality of non-transitory computer-readable storage media collectively store the complete code. In at least one embodiment, the executable instructions are executed such that different instructions are executed by different processors; for example, the non-transitory computer-readable storage media store the instructions, and the main central processing unit (“CPU”) executes some instructions while the graphics processing unit (“GPU”) executes other instructions. In at least one embodiment, different components of the computer system have separate processors, and the different processors execute different subsets of the instructions.

[0254] Therefore, in at least one embodiment, the computer system is configured to implement one or more services that perform the processes described herein individually or collectively, and such a computer system is configured with suitable hardware and / or software to enable the performance of the operations. Furthermore, the computer system implementing at least one embodiment of this disclosure is a single device, and in another embodiment it is a distributed computer system comprising multiple devices operating differently, such that the distributed computer system performs the operations described herein, and that no single device performs all operations.

[0255] The use of any and all examples or exemplary language (e.g., “such as”) provided herein is intended only to better illustrate embodiments of this disclosure and does not impose a limitation on the scope of the disclosure unless otherwise required. No language in the specification should be construed as indicating that any unclaimed element is essential to the practice of the disclosure.

[0256] Unless otherwise expressly stated, it will be understood that throughout this specification, terms such as “processing,” “calculation,” “operation,” “determine,” etc., refer to the actions and / or processes of a computer or computing system or similar electronic computing device that manipulate and / or convert data represented as physical quantities (e.g., electronic quantities) in the registers and / or memory of the computing system into other data similarly represented as physical quantities in the memory, registers, or other such information storage, transmission, or display devices of the computing system.

[0257] Similarly, the term "processor" can refer to any device or part of a device that processes electronic data from registers and / or memory and converts that electronic data into other electronic data that can be stored in registers and / or memory. As a non-limiting example, a "processor" can be a CPU or a GPU. A "computing platform" can include one or more processors. As used herein, a "software" process can include, for example, software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Likewise, each process can refer to multiple processes that execute instructions sequentially or in parallel, continuously or intermittently. The terms "system" and "method" are used interchangeably herein, provided that a system can embody one or more methods, and a method can be considered a system.

[0258] In this document, reference may be made to obtaining, acquiring, receiving, or inputting analog or digital data into a subsystem, computer system, or computer-implemented machine. Obtaining, acquiring, receiving, or inputting analog and digital data can be accomplished in various ways, such as by receiving data as a parameter to a function call or a call to an application programming interface (API). In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data can be accomplished by transmitting data via a serial or parallel interface. In another embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data can be accomplished by transmitting data from a providing entity to an acquiring entity via a computer network. Reference may also be made to providing, outputting, transmitting, sending, or presenting analog or digital data. In various examples, the process of providing, outputting, transmitting, sending, or presenting analog or digital data can be implemented by transmitting data as an input or output parameter to a function call, an API, or an inter-process communication mechanism.

[0259] While the discussion above illustrates example implementations of the described technologies, other architectures can be used to implement the described functionality and are intended to fall within the scope of this disclosure. Furthermore, although specific assignments of responsibilities have been defined above for discussion purposes, various functions and responsibilities can be assigned and divided in different ways depending on the circumstances.

[0260] Furthermore, although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter claimed in the appended claims is not necessarily limited to the specific features or actions described. Rather, specific features and actions are disclosed as exemplary forms for implementing the claims. The subject matter of this disclosure has been described in detail herein to satisfy statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors have envisioned that the claimed subject matter may also be embodied in other ways to include steps different from or similar combinations of steps described herein in conjunction with other current or future techniques. Moreover, although the terms “step” and / or “block” may be used herein to imply different elements of the method employed, these terms should not be construed as implying any particular order among or between the various steps disclosed herein, unless the order of the steps is explicitly described.

[0261] Example text support

[0262] The disclosure of this application also includes the following numbered clauses:

[0263] Clause 1. One or more processors, including processing circuitry, said processing circuitry being configured to: process data representing LiDAR data generated using at least one or more neural networks (NNs) from a plurality of views to generate first extracted feature data in a first view and second extracted feature data in a second view of the plurality of views, based on at least one or more neural networks (NNs).

[0264] Clause 2. One or more processors according to Clause 1, wherein the processing circuitry is further configured to: generate combined extracted feature data, the combined extracted feature data combining the first extracted feature data in the first view with a projected representation of the second extracted feature data projected onto the first view.

[0265] Clause 3. One or more processors as described in Clause 1 or 2, wherein said processing circuitry is further configured to: process the extracted feature data of the combination based at least on said one or more NNs to generate one or more outputs.

[0266] Clause 4. One or more processors as described in Clause 1, 2 or 3, wherein said processing circuitry is further configured to: control one or more operations of the self-machine based at least on said one or more outputs.

[0267] Clause 5. One or more processors as described in Clauses 1, 2, 3 or 4, wherein the data representing at least one of the first view or the second view processed by the one or more NNs includes multiple channels corresponding to multiple echoes supported by the at least one LiDAR sensor.

[0268] Clause 6. One or more processors according to Clause 1, 2, 3 or 4, wherein the processing circuitry is further configured to: generate the combined extracted feature data based at least on projecting a plurality of frames representing the second extracted feature data of a sequential LiDAR scan from the second view to the first view.

[0269] Clause 7. One or more processors according to Clauses 1, 2, 3 or 4, wherein the second extracted feature data includes an intermediate representation extracted using the one or more NNs, wherein the processing circuitry is further configured to: project the intermediate representation from the second view onto the first view.

[0270] Clause 8. One or more processors according to Clause 1, 2, 3 or 4, wherein the processing circuitry is further configured to: process the second extracted feature data based at least on the one or more NNs to generate artifact data, the artifact data comprising multiple artifact classification data channels corresponding to multiple supported LiDAR echoes encoded by the second view.

[0271] Clause 9. One or more processors according to Clauses 1, 2, 3 or 4, wherein the one or more outputs represent one or more detected obstacles extracted based at least on the extracted feature data of the combination, and the one or more operations of the self-machine include path planning based at least on the one or more detected obstacles.

[0272] Clause 10. One or more processors according to Clauses 1, 2, 3 or 4, wherein the one or more outputs represent detected airworthiness space extracted based at least on the extracted feature data of the combination, and the one or more operations of the ego machine include path planning based at least on the detected airworthiness space.

[0273] Clause 11. One or more processors as described in Clauses 1, 2, 3 or 4, wherein the one or more outputs represent one or more detected weather or surface conditions extracted based at least on the extracted feature data of the combination, and the one or more operations of the self-machine include controlling the speed of the self-machine based at least on the one or more detected weather or surface conditions.

[0274] Clause 12. One or more processors according to Clause 1, 2, 3 or 4, wherein the processing circuitry is further configured to: generate the combined extracted feature data based at least on projecting instances of one or more caches representing the second extracted feature data of the current time slice and the second extracted feature data of one or more previous time slices from the second view to the first view.

[0275] Clause 13. One or more processors as described in Clauses 1, 2, 3, or 4, wherein said one or more processors are included in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing optical transmission simulation; a system for performing collaborative content creation of 3D assets; a system for performing deep learning operations; a system for performing remote operations; a system for performing real-time streaming; a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; a system implemented using edge devices; a system implemented using robots; a system for performing conversational AI operations; a system implementing one or more language models; a system implementing one or more large language models (LLMs); a system implementing one or more visual language models (VLMs); a system implementing one or more multimodal language models; a system for generating synthetic data; a system for generating synthetic data using AI; a system for performing one or more generative AI operations; a system containing one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

[0276] Clause 14. A method comprising: controlling one or more operations of an ego machine based at least on one or more outputs of one or more neural networks NN, said one or more outputs being generated at least based on extracted feature data processed by said one or more NNs, said combined extracted feature data being a combination of first extracted feature data extracted from a first view of LiDAR data and a projected representation of second extracted feature data extracted from a second view of said LiDAR data and projected onto the first view.

[0277] Clause 15. The method according to Clause 14, wherein the second extracted feature data is extracted based at least on multiple LiDAR echoes encoded in the respective channels of the second view of the LiDAR data.

[0278] Clause 16. The method according to Clause 14 further comprises: generating the combined extracted feature data based at least on a plurality of frames representing the second extracted feature data of sequential LiDAR scans projected from the second view onto the first view.

[0279] Clause 17. The method according to Clause 14, wherein the second extracted feature data includes intermediate representations extracted by the one or more NNs, the method further comprising: projecting the intermediate representations from the second view onto the first view.

[0280] Clause 18. The method according to Clause 14 further comprises: processing the second extracted feature data using the one or more NNs to generate artifact data, the artifact data comprising multiple artifact classification data channels corresponding to multiple supported LiDAR echoes encoded by the second view.

[0281] Clause 19. The method according to Clause 14 further comprises: generating the combined extracted feature data by projecting at least one or more cached instances of the second extracted feature data representing the current time slice and the second extracted feature data representing one or more previous time slices from the second view to the first view.

[0282] Clause 20. The method according to Clause 14, wherein said method is performed by at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulated operations; a system for performing digital twin operations; a system for performing optical transmission simulation; a system for performing collaborative content creation of 3D assets; a system for performing deep learning operations; a system for performing remote operations; a system for performing real-time streaming; a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; a system implemented using an edge device; a system implemented using a robot; a system for performing conversational AI operations; a system implementing one or more language models; a system implementing one or more large language models (LLMs); a system implementing one or more visual language models (VLMs); a system implementing one or more multimodal language models; a system for generating synthetic data; a system for generating synthetic data using AI; a system for performing one or more generative AI operations; a system containing one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

[0283] Clause 21. A system comprising: one or more processors configured to control one or more operations of an ego machine in a simulation rendered using one or more optical transport simulation algorithms, at least based on one or more outputs of one or more neural networks (NNs), the one or more outputs being generated at least based on extracted feature data combined by the processing of the one or more NNs, the combined extracted feature data being a combination of first extracted feature data extracted in a first view and a projected representation of second extracted feature data extracted in a second view and projected onto the first view.

[0284] Clause 22. The system pursuant to Clause 21, wherein the simulation is generated at least in part using a 3D content collaboration platform for 3D assets.

[0285] Clause 23. The system described in Clause 21, wherein the 3D content collaboration platform for 3D assets uses OpenUSD.

Claims

1. One or more processors, including processing circuitry, said processing circuitry being used to: The data of multiple views representing LiDAR data generated by at least one or more LiDAR sensors using a self-machined machine are processed based on at least one or more neural networks (NNs) to generate first extracted feature data in a first view and second extracted feature data in a second view of the multiple views. Generate combined extracted feature data, wherein the combined extracted feature data combines the first extracted feature data in the first view with the projected representation of the second extracted feature data projected onto the first view; At least one or more outputs are generated by processing the extracted feature data of the combination based on one or more neural networks; as well as The self-machine's operation is controlled based on at least one or more of the outputs.

2. The processor of claim 1, wherein the data representing at least one of the first view or the second view processed by the one or more NNs includes a plurality of channels corresponding to a plurality of echoes supported by the at least one LiDAR sensor.

3. The processor of claim 1 or more, wherein the processing circuitry is further configured to: generate the combined extracted feature data based at least on projecting a plurality of frames representing the second extracted feature data of sequential LiDAR scanning from the second view to the first view.

4. The processor of claim 1, wherein the second extracted feature data includes an intermediate representation extracted using the one or more NNs, wherein the processing circuitry is further configured to: project the intermediate representation from the second view onto the first view.

5. The processor of claim 1 or more, wherein the processing circuitry is further configured to: process the second extracted feature data based at least on the one or more NNs to generate artifact data, the artifact data comprising a plurality of artifact classification data channels corresponding to a plurality of supported LiDAR echoes encoded by the second view.

6. The processor of claim 1, wherein the one or more outputs represent one or more detected obstacles extracted based at least on the extracted feature data of the combination, and the one or more operations of the ego machine include path planning based at least on the one or more detected obstacles.

7. The processor of claim 1, wherein the one or more outputs represent detected airworthiness space extracted based at least on the extracted feature data of the combination, and the one or more operations of the ego machine include path planning based at least on the detected airworthiness space.

8. The processor of claim 1, wherein the one or more outputs represent one or more detected weather or surface conditions extracted based at least on the extracted feature data of the combination, and the one or more operations of the self-machine include controlling the speed of the self-machine based at least on the one or more detected weather or surface conditions.

9. The processor of claim 1 or more, wherein the processing circuitry is further configured to: generate the combined extracted feature data based at least on projecting instances of one or more caches representing the second extracted feature data of the current time slice and the second extracted feature data of one or more previous time slices from the second view to the first view.

10. The processor of claim 1 or more, wherein the processor is included in at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system used to perform simulation operations; Systems used to perform digital twin operations; A system for performing optical transmission simulation; A system for collaborative content creation of 3D assets; A system used to perform deep learning operations; A system used to perform remote operations; Systems used for performing real-time streaming; A system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; Systems implemented using edge devices; Systems implemented using robots; Systems used to perform conversational AI operations; A system that implements one or more language models; A system that implements one or more large language model LLMs; A system that implements one or more Visual Language Models (VLMs); A system that implements one or more multimodal language models; A system for generating synthetic data; Systems for generating synthetic data using AI; A system for performing one or more generative AI operations; A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

11. A method comprising: One or more operations of an ego machine are controlled based on at least one or more outputs of one or more neural networks (NNs), said one or more outputs being generated based at least on extracted feature data processed by said one or more NNs, said combined extracted feature data being a combination of first extracted feature data extracted from a first view of LiDAR data and a projected representation of second extracted feature data extracted from a second view of said LiDAR data and projected onto the first view.

12. The method of claim 11, wherein the second extracted feature data is extracted based at least on a plurality of LiDAR echoes encoded in the respective channels of the second view of the LiDAR data.

13. The method of claim 11, further comprising: The combined extracted feature data is generated by projecting multiple frames representing the second extracted feature data from the second view onto the first view, at least based on the second extracted feature data of sequential LiDAR scans.

14. The method of claim 11, wherein the second extracted feature data comprises an intermediate representation extracted by the one or more neural networks, the method further comprising: The intermediate representation is projected from the second view onto the first view.

15. The method of claim 11, further comprising: The extracted feature data is processed using one or more neural networks to generate artifact data, which includes multiple artifact classification data channels corresponding to multiple supported LiDAR echoes encoded by the second view.

16. The method of claim 11, further comprising: The combined extracted feature data is generated by projecting instances of the second extracted feature data representing the current time slice and one or more caches representing the second extracted feature data representing one or more previous time slices from the second view to the first view.

17. The method of claim 11, wherein the method is performed by at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system used to perform simulation operations; Systems used to perform digital twin operations; A system for performing optical transmission simulation; A system for collaborative content creation of 3D assets; A system used to perform deep learning operations; A system used to perform remote operations; Systems used for performing real-time streaming; A system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; Systems implemented using edge devices; Systems implemented using robots; A system for performing conversational AI operations; A system that implements one or more language models; A system that implements one or more large language model LLMs; A system that implements one or more Visual Language Models (VLMs); A system that implements one or more multimodal language models; A system for generating synthetic data; Systems for generating synthetic data using AI; A system for performing one or more generative AI operations; A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

18. A system comprising: One or more processors are configured to control one or more operations of an ego machine in a simulation rendered using one or more optical transmission simulation algorithms, based at least on one or more outputs of one or more neural networks (NNs), the one or more outputs being generated at least based on extracted feature data combined by the one or more NNs, the combined extracted feature data being a combination of first extracted feature data extracted in a first view and a projected representation of second extracted feature data extracted in a second view and projected onto the first view.

19. The system of claim 18, wherein the simulation is generated at least in part using a 3D content collaboration platform for 3D assets.

20. The system of claim 18, wherein the 3D content collaboration platform for 3D assets uses OpenUSD.

Images