Voxel generation technique

Abstract

Devices, systems, and techniques convert a point cloud into voxels. In at least one embodiment, a processor makes the point cloud representation of the environment into voxels based at least partially on a data structure that uses point locations to represent the voxels.

Description

Technology Field

[0001] At least one embodiment relates to processing resources used to represent point clouds as voxels. At least one embodiment relates to processors or computing systems for point clouds for generating voxels based at least partially on one or more point locations and displaying the voxels in one or more data structures. Background Technology

[0002] Generating voxel representations of objects based on point cloud representations of these objects requires the use of significant computing resources. The use of these computing resources can be improved. Brief explanation of the drawing

[0003] FIG. 1 illustrates a block diagram of a system for generating a voxel representation of an environment based on a point cloud representation of the environment according to at least one embodiment. FIG. 2 illustrates a block diagram of a system for generating a hash table used to convert a point cloud into voxels, according to at least one embodiment. FIG. 3 illustrates a block diagram of a system for calculating an average feature value for each voxel in a voxel grid based at least partially on a hash table, according to at least one embodiment. FIG. 4 illustrates a process using one or more processors to perform operations to generate a voxel representation of an environment based on a point cloud representation, according to at least one embodiment. FIG. 5 illustrates a process of an application programming interface (API) function that enables one or more processors, according to at least one embodiment, to perform operations to generate a voxel representation of an environment based on a point cloud representation. FIG. 6 illustrates a block diagram of a driver and / or runtime used to enable one or more processors, according to at least one embodiment, to perform API functions that generate a voxel representation of an environment based on a point cloud representation. FIG. 7a illustrates logic according to at least one embodiment. FIG. 7b illustrates logic according to at least one embodiment. FIG. 8 illustrates the training and deployment of a neural network according to at least one embodiment. FIG. 9 illustrates an exemplary data center system according to at least one embodiment. FIG. 10a illustrates an example of an autonomous vehicle according to at least one embodiment. FIG. 10b illustrates examples of camera positions and fields of view for the autonomous vehicle of FIG. 10a according to at least one embodiment. FIG. 10c is a block diagram illustrating an exemplary system architecture for an autonomous vehicle of FIG. 10a according to at least one embodiment. FIG. 10d is a drawing illustrating a system for communication between the cloud-based server(s) of FIG. 10a and an autonomous vehicle according to at least one embodiment. FIG. 11 is a block diagram illustrating a computer system according to at least one embodiment. FIG. 12 is a block diagram illustrating a computer system according to at least one embodiment. FIG. 13 illustrates a computer system according to at least one embodiment. FIG. 14 illustrates a computer system according to at least one embodiment. FIG. 15a illustrates a computer system according to at least one embodiment. FIG. 15b illustrates a computer system according to at least one embodiment. FIG. 15c illustrates a computer system according to at least one embodiment. FIG. 15d illustrates a computer system according to at least one embodiment. FIGS. 15e and FIGS. 15f illustrate a shared programming model according to at least one embodiment. FIG. 16 illustrates exemplary integrated circuits and associated graphics processors according to at least one embodiment. FIGS. 17a and FIGS. 17b illustrate exemplary integrated circuits and associated graphics processors according to at least one embodiment. FIGS. 18a and FIGS. 18b illustrate additional exemplary graphics processor logic according to at least one embodiment. FIG. 19 illustrates a computer system according to at least one embodiment. FIG. 20a illustrates a parallel processor according to at least one embodiment. FIG. 20b illustrates a partition unit according to at least one embodiment. FIG. 20c illustrates a processing cluster according to at least one embodiment. FIG. 20d illustrates a graphics multiprocessor according to at least one embodiment. FIG. 21 illustrates a multi-GPU (graphics processing unit) system according to at least one embodiment. FIG. 22 illustrates a graphics processor according to at least one embodiment. FIG. 23 is a block diagram illustrating a processor microarchitecture for a processor according to at least one embodiment. FIG. 24 illustrates a deep learning application processor according to at least one embodiment. FIG. 25 is a block diagram illustrating an exemplary neuromorphic processor according to at least one embodiment. FIG. 26 illustrates at least a portion of a graphics processor according to one or more embodiments. FIG. 27 illustrates at least a portion of a graphics processor according to one or more embodiments. FIG. 28 illustrates at least a portion of a graphics processor according to one or more embodiments. FIG. 29 is a block diagram of a graphics processing engine of a graphics processor according to at least one embodiment. FIG. 30 is a block diagram of at least a portion of a graphics processor core according to at least one embodiment. FIGS. 31a and FIGS. 31b illustrate thread execution logic including an array of processing elements of a graphics processor core according to at least one embodiment. FIG. 32 illustrates a "PPU (parallel processing unit)" according to at least one embodiment. FIG. 33 illustrates a "general processing cluster (GPC)" according to at least one embodiment. FIG. 34 illustrates a memory partition unit of a "PPU (parallel processing unit)" according to at least one embodiment. FIG. 35 illustrates a streaming multi-processor according to at least one embodiment. FIG. 36 is an exemplary data flow diagram for an advanced computing pipeline according to at least one embodiment. FIG. 37 is a system diagram of an exemplary system for training, adapting, instantiating, and deploying a machine learning model in an advanced computing pipeline according to at least one embodiment. FIG. 38 includes an exemplary example of an advanced computing pipeline (3710A) for processing imaging data according to at least one embodiment. FIG. 39a includes an exemplary data flow diagram of a virtual device supporting an ultrasonic device according to at least one embodiment. FIG. 39b includes an exemplary data flow diagram of a virtual device supporting a CT scanner according to at least one embodiment. FIG. 40a illustrates a data flow diagram for a process for training a machine learning model according to at least one embodiment. FIG. 40b is an exemplary example of a client-server architecture that enhances annotation tools with pre-trained annotation models according to at least one embodiment. FIG. 41 illustrates components of a system for accessing a large-scale language model according to at least one embodiment. Specific details for implementing the invention

[0004] In the following description, a number of specific details are presented to provide a more thorough understanding of at least one embodiment. However, a person skilled in the art will clearly understand that the concepts of the present invention may be practiced without one or more of these specific details, and that any two or more aspects of any one or more embodiments described herein may be combined.

[0005] In at least one embodiment, one or more processors comprising one or more circuits perform operations to generate one or more voxels using one or more point clouds based on one or more point locations, at least partially, and to display one or more voxels in one or more data structures. In at least one embodiment, one or more processors generate voxels by displaying voxels on a display device or otherwise activating them. In at least one embodiment, one or more data structures include one or more hash tables that correlate one or more point locations with one or more of the representations of one or more voxels to be generated, and are further described herein together with at least the voxelization hash table (208) of FIG. 2. In at least one embodiment, a voxel is a representation unit of three-dimensional (3D) space used in computer graphics and computer vision applications. In at least one embodiment, a voxel is referred to as a 3D pixel. In at least one embodiment, a voxel is visually represented as a cube. In at least one embodiment, voxelization refers to one or more techniques used to convert a point cloud representation of an environment and / or object into a voxel representation of that environment and / or object. In at least one embodiment, one or more processors calculate the average feature value of a voxel by using the hash table described herein to display the voxels to be generated on a display device without checking the feature values ​​at memory locations corresponding to the voxels not to be generated. In at least one embodiment, the feature value is a value used at least partially to generate the visual feature of the voxel, and is further described herein.

[0006] In at least one embodiment, one or more processors use one or more locations (point locations) of points in a voxel grid stored in one or more hash tables to indicate voxels to be generated to represent one or more points of one or more point clouds. In at least one embodiment, the voxel grid is a three-dimensional data structure containing voxels. In at least one embodiment, one or more processors store each point location in a hash table so that a point location retrieved from the hash table can indicate a corresponding voxel to be generated. In at least one embodiment, one or more processors create one or more hash tables so that one or more processors can efficiently retrieve one or more indications of one or more voxels to be generated using one or more voxel locations. In at least one embodiment, one or more processors create one or more hash tables so that when the number of voxels in the voxel grid exceeds the number of points in the point cloud, one or more processors can efficiently identify one or more voxels representing one or more points. In at least one embodiment, one or more processors generate one or more hash tables that allow one or more processors to generate one or more feature values ​​of one or more voxels by iterating at least partially for each of one or more point locations in addition to or instead of iterating over each possible voxel in a voxel grid. In at least one embodiment, one or more processors use the hash tables to identify point feature values ​​common to common voxels and enable these processors to calculate one or more feature values ​​of voxels, such as sum feature values ​​and average feature values ​​further described herein.In at least one embodiment, one or more processors voxelize a point cloud using a hash table as part of a process for downsampling the resolution of the point cloud and using the voxel representation of the point cloud as inputs in neural network operations such as image classification, image segmentation, autonomous driving, or some combination thereof. In at least one embodiment, reference is made to techniques used to convert a point cloud into voxels to voxelize the point cloud.

[0007] FIG. 1 illustrates a block diagram of a system (100) comprising one or more processors, comprising one or more circuits that generate one or more voxels using one or more point clouds based at least partially on one or more point locations and display one or more voxels in one or more data structures. In at least one embodiment, one or more aspects of one or more embodiments described herein with FIG. 1 are combined with one or more aspects of one or more embodiments described herein, including those described with at least FIG. 2 through 6. In at least one embodiment, one or more processors perform one or more operations of the system (100). In at least one embodiment, one or more processors performing one or more operations of the system (100) are any one processor or combination of processors described herein, comprising a CPU (1302) described with FIG. 13, accelerator(s) (1014) described with FIG. 10c, a graphics processor (1710) described with FIG. 17a, and a parallel processing unit ("PPU") (3200) described with FIG. 32. In at least one embodiment, the processor(s) (102) perform operations used by the system (100), such as loading / storing values ​​output by neural network activation functions in arithmetic logic unit(s) (ALUs), such as the ALU(s) (710) of FIG. 7. In at least one embodiment, the processor(s) (102) perform one or more operations described with FIG. 2, such as generating a voxelized hash table (208). In at least one embodiment, the processor(s) (102) perform one or more operations described with FIG. 3, such as calculating the number of points in a voxel. In at least one embodiment, the processor(s) (102) perform one or more operations described with FIG. 4, and calculate voxel offsets with these operations 402.In at least one embodiment, the processor(s) (102) perform one or more operations described together with FIG. 5, such as performing voxelization API functions as operation 504. In at least one embodiment, the processor(s) (102) perform one or more operations described together with FIG. 6, such as operations of API(s) (610).

[0008] In at least one embodiment, the system (100) is any computing system such as an edge computing system, an accelerated computing system, a high-performance computing system, a data center, a cloud computing system, or a combination of some of these. In at least one embodiment, the system (100) is used in fields such as healthcare, genomics, engineering, aerospace, urban planning, graphics processing, finance, data storage and management, online commerce, meteorology, physics modeling, or a combination of some of these. In at least one embodiment, the system (100) is used to perform artificial intelligence (AI) tasks such as image classification, image segmentation, autonomous driving, manufacturing defect identification, or a combination of some of these.

[0009] In at least one embodiment, the system (100) includes sensor(s) (101). In at least one embodiment, the sensor(s) (101) collect data from an environment to be used by a processor and generate a point cloud representation of the environment, including objects in the environment. In at least one embodiment, the point cloud representation is a 3D representation of the environment. In at least one embodiment, one or more sensor(s) (101) are a 3D scanner, a 3D sensor, a light detection and distance measurement (LIDAR or LiDAR) sensor, or a combination of some of these. In at least one embodiment, one or more sensor(s) (101) are used in fields such as autonomous driving, manufacturing defect detection, object identification, or a combination of some of these.

[0010] In at least one embodiment, the system (100) includes processor(s) (102). In at least one embodiment, the processor(s) (102) is any one processor or combination of processors described herein, including a CPU (1302) described with FIG. 13, an accelerator(s) (1014) described with FIG. 10c, a graphics processor (1710) described with FIG. 17a, and a parallel processing unit ("PPU") (3200) described with FIG. 32. In at least one embodiment, the processor(s) (102) is a processor implemented in an edge computing system designed to perform AI tasks such as image classification, autonomous driving, or some combination thereof. In at least one embodiment, the processor(s) (102) is an AMD® Epyc™ embedded processor and / or NVIDIA® Jetson™ TX2 module comprising a plurality of types of processors.

[0011] In at least one embodiment, as used in any implementation described herein, terms such as “system,” “device,” “component,” or “module,” and nominalized verbs (e.g., compiler, and / or other terms), each refer to any combination of software logic, firmware logic, hardware logic, and / or circuits configured to provide the functionality described herein. In at least one embodiment, any combination of software logic, firmware logic, hardware logic, and / or circuits configured to provide the functionality described herein is referred to as a component. In at least one embodiment, any component described herein is combined with and / or communicably connected to at least one other component, regardless of how such components are described as being combined and / or communicably connected in other embodiments. In at least one embodiment, software may be implemented as a software package, code, and / or instruction set or instructions. In at least one embodiment, the hardware comprises, alone or in any combination, a hardwired circuit, a programmable circuit, a state machine circuit, a fixed function circuit, an execution unit circuit, and / or firmware that stores instructions executed by the programmable circuit. In at least one embodiment, the modules may be implemented collectively or individually as circuits forming part of a larger system, e.g., an integrated circuit (IC), a system-on-chip (SoC), etc. In at least one embodiment, any one or more architectures of any circuit of any one or more modules are represented as a register-transfer level (RTL) representation and / or other fabless representation that can be licensed and / or used at tape-out, which is the final phase of IC design before being used to manufacture the IC.

[0012] In at least one embodiment, one or more processors comprising one or more circuits perform a voxelization module (104) to generate one or more voxels using one or more point clouds based at least partially on one or more point locations and to represent one or more voxels in one or more data structures. In at least one embodiment, the voxelization module (104) outputs a voxel-based representation of point cloud data. In at least one embodiment, the voxelization module (104) performs voxelization of point cloud data (103) (data of the point cloud) using a voxelization hash table, a type of hash table further described herein together with at least the voxelization hash table (208) of FIG. 2. In at least one embodiment, the voxelization module (104) performs operations to perform downsampling to a less data-intensive voxel representation of the point cloud. In at least one embodiment, the voxelization module (104) downsamples point cloud data (103) into voxel data, enabling processor(s) (102) to perform AI tasks, such as image classification, with less latency. In at least one embodiment, the voxelization module (104) is an exemplary module capable of performing any one or more operations used to voxelize a point cloud representation of an environment by using a voxelization hash table as further described herein, at least partially, together with the voxelization hash table (208) of FIG. 2. In at least one embodiment, a voxel containing one or more points or point locations refers to a voxel that must be created, displayed, or otherwise activated to represent these one or more points. In at least one embodiment, a point or point location contained in a voxel is referred to as being assigned to the voxel.In at least one embodiment, a voxel containing one or more points refers to a voxel encompassing one or more points, such that, at least conceptually, a point cloud is positioned to overlap with a voxel grid. In at least one embodiment, a voxel containing points is a voxel whose boundaries overlap with the points.

[0013] In at least one embodiment, the voxelization module (104) allocates an amount of memory based on the number of points in the point cloud data (103) to generate a voxelization hash table, which is further described herein together with at least the voxelization hash table (208) of FIG. 2. In at least one embodiment, the amount of memory includes a plurality of memory locations. In at least one embodiment, the voxelization module (104) calculates the locations of each point in the point cloud based on the coordinates of each point. In at least one embodiment, the locations of the points are referred to as point locations. In at least one embodiment, a point location is referred to as a voxel offset. In at least one embodiment, a voxel offset is one or more values ​​indicating the location of a voxel within a voxel grid. In at least one embodiment, a voxel offset is a set of three coordinates in a 3D coordinate system. In at least one embodiment, a voxel offset is a single value based on the set of three coordinates. In at least one embodiment, a voxel offset represents a point location by using a distance and direction from a location within the voxel grid, such as the origin (0,0,0). In at least one embodiment, the voxelization module (104) uses a voxel offset to identify a voxel identifier (voxel ID) for a voxel containing the voxel offset. In at least one embodiment, the voxel ID is an identifier (indicator) of a specific voxel within a voxel grid.

[0014] In at least one embodiment, the processor performs a voxelization module (104) to receive and / or otherwise acquire point cloud data (103). In at least one embodiment, as further described herein with at least FIG. 2, the voxel offset and voxel ID of each point in the point cloud data (103) are input into a hash table used to identify voxels using each point. In at least one embodiment, the voxelization module (104) sums all point features within the voxel and outputs the sum, as further described herein with at least FIG. 3 and FIG. 4. In at least one embodiment, the voxelization module (104) calculates an average feature value for each voxel, as further described herein with at least FIG. 3. In at least one embodiment, the voxelization module (104) outputs the number of points and the average feature value for each voxel to be used for further processing, as further described herein with at least FIG. 3.

[0015] In at least one embodiment, a neural network (NN) classification module (106) receives and / or otherwise obtains a voxel-based representation of point cloud data generated by a voxelization module (104). In at least one embodiment, the neural network classification module (106) receives and / or otherwise obtains input data including a voxel representation of the environment based at least partially on a data structure to indicate the voxels to be generated during the conversion to the corresponding voxel representation of the point cloud, as described herein together with FIGS. 1 through 6. In at least one embodiment, the input data is a voxel representation of a 3D medical image. In at least one embodiment, the neural network classification module (106) includes one or more neural networks trained to perform classification tasks. In at least one embodiment, the processor(s) (102) performs the neural network by performing one or more operations related to the neural network tasks, such as loading / storing weights during forward propagation or backward propagation, in an ALU(s) such as the ALU(s) (710) of FIG. 7. In at least one embodiment, the neural network is referred to as a neural network model, a neural model, or a model. In at least one embodiment, the neural network comprises one or more neural networks. In at least one embodiment, the neural network is a deep learning network or a deep neural network (DNN). In at least one embodiment, the neural network is a recurrent neural network (RNN), a convolutional neural network (CNN), a generative adversarial network (GAN), a transformer, a graph neural network (GNN), or a combination of some of these. In at least one embodiment, a training system uses training data to train an untrained neural network to produce a trained neural network, which is a neural network that performs inferences or predictions at a level of accuracy above a threshold level of accuracy set by a user or application.In at least one embodiment, the neural network classification module (106) is for identifying and labeling objects in voxel-based data, such as a voxel representation of an environment detected by sensors installed on an autonomous vehicle.

[0016] FIG. 2 illustrates a block diagram of a system (200) comprising one or more processors, comprising one or more circuits for performing one or more operations to enable information regarding one or more point clouds and one or more voxel grids to be used to generate and store data in a voxelization hash table as part of a voxelization process. In at least one embodiment, one or more aspects of one or more embodiments described herein together with FIG. 2 are combined with one or more aspects of one or more embodiments described herein, including those described together with FIG. 1 and FIG. 3 through 6. In at least one embodiment, the system (200) comprises one or more processors for performing one or more operations of the system (200). In at least one embodiment, one or more processors performing one or more operations of the system (200) are any one processor or combination of processors described herein, including a CPU (1302) described with FIG. 13, accelerator(s) (1014) described with FIG. 10c, a graphics processor (1710) described with FIG. 17a, and a parallel processing unit ("PPU") (3200) described with FIG. 32. In at least one embodiment, the system (200) includes one or more modules described herein, such as the neural network classification module (106) of FIG. 1. In at least one embodiment, one or more operations of the system (200) are combined with one or more operations of the system (300) of FIG. 3, such as calculating the number of points per voxel. In at least one embodiment, one or more operations of the system (200) are combined with one or more operations of the process (400) of FIG. 4, such as allocating memory locations as operation 401. In at least one embodiment, one or more operations of the system (200) are combined with one or more operations of the process (500) of FIG. 5, such as inputting point cloud data into the API(s) as operation 502.In at least one embodiment, one or more operations of the system (200) are one or more operations of the API(s) (610), such as calculating the number of points per voxel using the voxelization module (304).

[0017] In at least one embodiment, the system (200) includes one or more components of the system (100) of FIG. 1. In at least one embodiment, the system (200) is an edge computing device designed to perform AI tasks, such as medical segmentation or autonomous driving. An edge computing system is one or more computing systems in which proximity to other computing systems, such as servers and databases, improves various aspects of computing performance, such as response times and reliability of data transmissions. In at least one embodiment, the edge computing system is part of a distributed computing framework, wherein a plurality of computing systems are physically separated and connected by a network such as a cloud computing network. In at least one embodiment, the edge computing system is a computing system having an NVIDIA® Jetson™ computing system or an AMD® Epyc™ embedded processor. In at least one embodiment, the edge computing system performing AI-related operations is referred to as an AI hardware platform. In at least one embodiment, the system (200) includes one or more sensor(s) (101) of FIG. 1, such as camera sensors, radar sensors, lidar sensors, laser sensors, and ultrasonic sensors. In at least one embodiment, the system (200) includes one or more neural network classification modules (106).

[0018] In at least one embodiment, the system (200) includes a voxelization module (204). In at least one embodiment, the voxelization module (204) includes one or more components of the voxelization module (104). In at least one embodiment, the voxelization module (204) performs one or more operations of the voxelization module (104). In at least one embodiment, the voxelization module (204) receives and / or otherwise acquires point cloud data (103) based at least partially on one or more sensors, such as the sensor(s) (101) of FIG. 1. In at least one embodiment, the voxelization module (204) outputs voxel offsets and voxel index values ​​to be loaded / stored in a voxelization hash table as further described herein.

[0019] In at least one embodiment, the system (200) has voxel offsets (205a- n Includes one or more point locations such as ), where n represents the number of voxel offsets. In at least one embodiment, the voxelization module (204) comprises voxel offsets (205a- n Generates and outputs ). In at least one embodiment, a voxel offset is one or more values ​​representing the position of a point in a corresponding voxel within a voxel grid. In at least one embodiment, a voxel offset is one or more values ​​representing the position of a point in a voxel grid used by a processor to identify a specific voxel where the point is located. In at least one embodiment, the voxel offset is based at least partially on three coordinate values ​​such as x, y, and z in a 3D coordinate system. In at least one embodiment, voxel offsets (205a- n) is based on the center point of the voxel. In at least one embodiment, the voxel offset value represents the distance from a point within the voxel grid, such as the origin (0,0,0). In at least one embodiment, the voxel offset value alternatively represents the location of a point within the voxel by indicating the distance of that point from a given location within the voxel, such as the center of the voxel, which is used to calculate the center of two or more points contained in the voxel. In at least one embodiment, the center of two or more points contained in the voxel is used to calculate the features of the voxel.

[0020] In at least one embodiment, the voxelization module (204) performs operations such as voxelization function(s) using the voxelization hash table (612) of FIG. 6 (function(s)(612)) to correlate each point of the point cloud to a voxel within the voxel grid, wherein such a function is identified in pseudocode having a name such as map_point_to_voxel(). In at least one embodiment, the voxelization module (204) performs map_point_to_voxel() to calculate the locations of the voxels based on the total grid size, the number of voxels, the voxel grid resolution, the dimensions of each voxel, or some combination thereof. In at least one embodiment, as further described herein, the voxelization module (204) creates a data structure and indexes each of one or more point locations of the data structure to one or more voxels of the data structure. In at least one embodiment, the voxelization module (204) indexes point locations to voxel IDs using a hash function as further described herein. In at least one embodiment, another method of indexing point locations to voxels loads / stores point locations in array locations and loads / stores corresponding voxel IDs in corresponding locations in other arrays.

[0021] In at least one embodiment, the voxelization module (204) assigns a unique voxel ID value to each voxel of the voxel grid. In at least one embodiment, the voxelization module (204) determines a voxel offset for each point of the point cloud and uses the voxel offset to identify a specific voxel having a voxel ID. In at least one embodiment, the voxelization module (204) has voxel offsets (205a- n Voxel IDs corresponding to specific voxels containing ) (206a- n ) outputs and / or stores to data storage locations, such as buffers or caches, which are indexed by these locations. In at least one embodiment, the index of data storage locations storing voxel IDs is referred to as a voxel index and consists of voxel index values. In at least one embodiment, at least conceptually, the voxelization module (204) includes voxel offsets (205a- n A voxel ID for each point within a voxel is identified based on the voxel offset of ). In at least one embodiment, 3s are any real numbers of any data format, such as FP16, suitable for uniquely representing each voxel of a voxel grid.

[0022] In at least one embodiment, the voxelization module (204) includes voxel offsets (205a-) further described herein, which are to be stored in a voxelization hash table (208). n ) and voxel index values(206a- nOutputs ). In at least one embodiment, the voxelized hash table (208) is referred to as a hash map, lookup table, or mapping. In at least one embodiment, the data of the voxelized hash table (208) is loaded / stored in a data storage device (storage device), also referred to as memory or memory device. In at least one embodiment, the data of the voxelized hash table (208) is loaded / stored in memory, such as cache memory, random access memory (RAM), dynamic random addressable memory (DRAM), static random addressable memory (SRAM), non-volatile memory (e.g., flash memory), other data storage, or some combination thereof. In at least one embodiment, the data of the voxelized hash table (208) is loaded / stored in a data storage that is any one or a combination of the data storage devices described herein, including the data storage (701) of FIG. 7.

[0023] In at least one embodiment, the voxelization module (204) creates data structures including hash tables that store point locations as keys and represent corresponding voxels as corresponding values, wherein the voxels are created, displayed, or activated to represent one or more points represented by the point locations. In at least one embodiment, the voxelization module (204) creates a voxelization hash table by using and / or allocating an amount of memory based on the number of point locations or points. In at least one embodiment, the voxelization module (204) creates a voxelization hash table by allocating, in a storage device, an amount of memory that is twice the number of points in the point cloud that stores the voxel offset of each point, and a corresponding voxel index value that identifies the memory location that stores the voxel ID. In at least one embodiment, the voxelization hash table stores the corresponding voxel ID instead of the voxel index value.

[0024] In at least one embodiment, the voxelization hash table (308) stores voxel locations (voxel offsets) as keys and voxel index values ​​or voxel IDs as values ​​corresponding to those keys. In at least one embodiment, the voxelization module (204) assigns a different key (unique value) to each point in the point cloud as a method of identifying each point. In at least one embodiment, the key assigned to each point in the point cloud is the voxel offsets (205a- n It is a voxel offset of ). In at least one embodiment, a key is input to a hash function and outputs a unique code for the key, a hash value indicating a memory location where a value corresponding to the key is stored. In at least one embodiment, the hash function is a function of any type that maps an input value to another value, wherein the output of the function indicates where the other value is located in memory. In at least one embodiment, the hash function uses any type of hashing method, such as division, middle-square, folding, multiplication, or some combination thereof. In at least one embodiment, one or more indications of one or more point locations are input to the hash function as keys and output one or more indications of one or more voxels, such as voxel index values ​​or voxel IDs. In at least one embodiment, one or more operations of inputting a key to the hash function to generate an output are represented by arrows in the voxelized hash table (208). In at least one embodiment, voxel offsets (205a- n One of ) is the voxel index values ​​(210a- n It is a key input to a hash function to output ).

[0025] In at least one embodiment, the voxelization module (204) generates data structures including hash tables based on one or more arrays of one or more point locations and one or more arrays of one or more representations of one or more voxels. In at least one embodiment, the voxelization module (204) loads / stores data as one or more arrays in the voxelization hash table (308). In at least one embodiment, voxel offsets (205a- n The array maintaining ) is an array of length equal to the number of points in the point cloud. In at least one embodiment, voxel index values ​​(210a- n The array that maintains ) is an array of length equal to the number of points in the point cloud.

[0026] In at least one embodiment, the voxelization module (204) identifies valid voxels, which are voxels to be created, displayed, or activated on a display device. In at least one embodiment, a valid voxel is a voxel containing a number of points that satisfies or exceeds a threshold number of points. In at least one embodiment, for example, when the threshold number of points is 5, a valid voxel is a voxel containing 5 or more points. In at least one embodiment, the valid voxel number refers to the voxel ID of the valid voxel.

[0027] In at least one embodiment, voxel index values ​​(210a- n Each voxel index value of ) is the voxel offsets (205a- n Voxel IDs corresponding to the voxel offset of ) (206a- n Indicates a memory location storing the voxel ID of ). In at least one embodiment, the voxel index value indicates a location or index number in an array where a data value is stored, and the location is used to access data at the corresponding memory location. In at least one embodiment, voxel index values ​​(210a- nEach of ) is a string of numbers and / or characters. In at least one embodiment, voxel index values ​​(210a- n Each of these is not the output of a hash function, but a voxel ID identified by a voxelization module (204) to correlate with a point location as further described herein. In at least one embodiment, the voxelization module (204) receives and / or otherwise obtains data from a voxelization hash table (208) to be used to determine the number of points in each voxel and / or the average feature value of each voxel, at least partially, which is further described herein together with at least FIG. 3.

[0028] FIG. 3 illustrates a block diagram of a system (300) comprising one or more processors including one or more circuits for performing one or more operations that cause the calculation of the number of points per voxel and / or the calculation of the average feature value per voxel using a voxelized hash table according to at least one embodiment. In at least one embodiment, one or more aspects of one or more embodiments described herein together with FIG. 3 are combined with one or more aspects of one or more embodiments described herein, including those described together with FIG. 1 to 2 and FIG. 4 to 6. In at least one embodiment, the system (300) comprises one or more processors for performing one or more operations of the system (300). In at least one embodiment, one or more processors performing one or more operations of the system (300) are any one processor or combination of processors described herein, including a CPU (1302) described with FIG. 13, accelerator(s) (1014) described with FIG. 10c, a graphics processor (1710) described with FIG. 17a, and a parallel processing unit ("PPU") (3200) described with FIG. 32. In at least one embodiment, the system (300) includes one or more modules described herein, such as the neural network classification module (106) of FIG. 1. In at least one embodiment, one or more operations of the system (300) are combined with one or more operations of the system (200) of FIG. 2, such as calculating voxel offsets. In at least one embodiment, one or more operations of the system (300) are combined with one or more operations of the process (400) of FIG. 4, such as allocating memory locations to operation 401. In at least one embodiment, one or more operations of the system (300) are combined with one or more operations of the process (500) of FIG. 5, such as inputting point cloud data into the API(s) as operation 502.In at least one embodiment, one or more operations of the system (300) are one or more operations of the API(s) (610), such as calculating the number of points per voxel using the voxelization module (304).

[0029] In at least one embodiment, the voxelized hash table (308) is the voxelized hash table (208), which is described in more detail herein together with at least FIG. 2. In at least one embodiment, the voxelized hash table (308) is any data structure that correlates points of a point cloud with voxel IDs, such as a linked list, a tree graph, a multidimensional tensor, or some combination thereof. In at least one embodiment, using the voxelized hash table (308) to correlate data values ​​is also referred to as indexing, associating, or linking these data values.

[0030] In at least one embodiment, voxel offsets (305a- n ) are voxel offsets (205a- n ) and is further described herein together with at least FIG. 2. In at least one embodiment, voxel offsets (305a- n The number of ) is equal to the number of points in the point cloud. In at least one embodiment, the data voxel offsets (305a-) of the voxelized hash table (308) n The amount of memory used to store ) is equal to the number of points in the point cloud.

[0031] In at least one embodiment, voxel index values ​​(310a- n ) are voxel index values ​​(310a- n ) and is further described herein together with at least FIG. 2. In at least one embodiment, voxel index values ​​(310a- nThe number of ) is equal to the number of points in the point cloud. In at least one embodiment, the data voxel index values ​​(310a-) of the voxelized hash table (308) n The amount of memory used to store ) is equal to the number of points in the point cloud.

[0032] In at least one embodiment, the voxelization module (304) is the voxelization module (204), which is further described herein together with at least FIG. 2. In at least one embodiment, the voxelization module (304) is the voxel offsets (205a-) of FIG. 2. n ) and voxel IDs(206a- n ) is generated. In at least one embodiment, for each point of the point cloud, the voxelization module (304) identifies a specific voxel ID of the voxel containing the point by using a voxel offset (key) representing the point to look up the corresponding voxel index (hash value or corresponding value) of the voxelization hash table.

[0033] In at least one embodiment, the voxelization module (304) calculates the number of points per voxel in a voxel grid. In at least one embodiment, the voxelization module (304) allocates memory locations to load / store the number of points in each voxel, and these memory locations are collectively referred to as the output space and are represented as output_space in pseudocode. In at least one embodiment, voxel offsets (305a- n For each point represented by one of ), the voxelization module (304) represents the voxel index values ​​(310a- n ) is used to identify the corresponding voxel ID of the voxel containing the corresponding point. In at least one embodiment, the voxelization module (304) identifies points and / or voxel offsets (305a-) for each voxel ID identified in the voxelization hash table (308). nCounts the number of ). In at least one embodiment, points and / or voxel offsets (305a-) associated with each voxel ID. n By counting the number of ), the voxelization module (304) obtains the voxels (312a- n It can generate and / or output the number of points for each voxel, such as the number of points per ) In at least one embodiment, the voxelization module (304) outputs the voxels (312a-) in the output space. n Loads / stores the number of points per ) In at least one embodiment, a voxel (312a- n The number of points per ) is, at least partially, a voxel (314a- n It is used to calculate average feature values ​​per )

[0034] In at least one embodiment, the voxelization module (304) identifies data values ​​used to perform one or more operations to generate one or more feature values ​​of one or more voxels based on one or more data structures, such as a voxelization hash table (308). In at least one embodiment, the feature of a point is any characteristic of an object at that point, expressed as a numeric value. In at least one embodiment, the point cloud data is augmented, labeled, or encoded with one or more features assigned to each point. In at least one embodiment, the features of a point include reflectance, texture, color, or some combination thereof. In at least one embodiment, the average feature of a voxel is the average feature value of all points within that voxel. In at least one embodiment, for example, the average feature value of the intensity of a voxel is the average intensity value of all points within that voxel. In at least one embodiment, the feature of a point is location (place) information of that point within the voxel. In at least one embodiment, for example, the average feature of the point locations of a voxel is the average location of all points within the voxel. In at least one embodiment, the average feature of the color of a voxel is the average color value of all points within that voxel.

[0035] In at least one embodiment, the feature value of the voxel is an average feature value also referred to as an average feature. In at least one embodiment, the voxelization module (304) is a voxel (314a- nOne or more average feature values ​​are calculated per voxel. In at least one embodiment, the voxelization module (304) calculates average feature values ​​only for voxels that are valid voxels, using a voxelization hash table as further described herein with at least FIG. 2. In at least one embodiment, the average feature value per voxel is one or more values ​​representing the average feature value for each voxel containing a point. In at least one embodiment, the feature value is referred to as a feature. In at least one embodiment, the point cloud includes one or more features for each point in the point cloud. In at least one embodiment, the voxelization module (304) calculates the voxels (312a- n Using his calculations of the number of points per voxel(314a- n ) Calculates average feature values ​​per voxel. In at least one embodiment, the voxelization module (304) calculates the average feature value of a voxel by summing all values ​​of the type of feature assigned to each point within the voxel together, and divides the sum by the total number of points within the voxel. In at least one embodiment, other averaging techniques, such as weighted averaging, are used. In at least one embodiment, the voxelization module (304) voxels (314a- n Load / store each value (average value) of the average feature values ​​per ) into the output space.

[0036] In at least one embodiment, a voxel (312a- n Number of points and voxels per ) n Values ​​loaded / stored in the output space, such as average feature values ​​per ) are further processed to output a voxel representation of the point cloud on the display. In at least one embodiment, for example, values ​​loaded / stored in the output space are used to display voxels having a specific color, texture, transparency, or some combination thereof.

[0037] FIG. 4 illustrates a block diagram of a process (400) according to at least one embodiment, which enables a processor to perform one or more operations for generating a voxel representation of a point cloud representation of an environment. In at least one embodiment, one or more aspects of one or more embodiments described herein together with FIG. 4 are combined with one or more aspects of one or more embodiments described herein, including those described together with FIG. 1 to 3 and FIG. 5 to 6. In at least one embodiment, one or more processors perform one or more operations of the process (400). In at least one embodiment, one or more processors performing one or more operations of process (400) are any one processor or combination of processors described herein, including processor(s) (102) described with FIG. 1, CPU (1302) described with FIG. 13, accelerator(s) (1014) described with FIG. 10c, graphics processor (1710) described with FIG. 17a, and parallel processing unit ("PPU") (3200) described with FIG. 32. In at least one embodiment, processor(s) (102) perform one or more operations of process (400), such as mapping points to voxels with operation 402. In at least one embodiment, one or more operations of process (400) are one or more operations of system (200) of FIG. 2, such as loading / storing voxel offsets of operation 404. In at least one embodiment, one or more operations of the process (400) are combined with one or more operations of the system (300) of FIG. 3, such as calculating feature average values ​​with operation 408. In at least one embodiment, one or more operations of the process (400) are combined with one or more operations of the process (500) of FIG. 5, such as inputting point cloud data into API(s) with operation 502.In at least one embodiment, one or more operations of the process (400) are one or more operations of the API(s) (610), such as loading / saving the voxel offsets of operation 404.

[0038] In at least one embodiment, the processor starts the process (400) with operation 401 by performing operations to allocate memory locations of one or more data storage devices. In at least one embodiment, allocating memory locations to the process (400) is referred to as initialization. In at least one embodiment, the processor allocates memory locations of one or more data storage devices to store data of one or more arrays, as further described herein. In at least one embodiment, the operations include allocating memory locations to store an array (or other tensor) of voxel grid resolution information, such as voxel size. In at least one embodiment, the array of voxel grid resolution information includes three-dimensional values ​​such as the height, width, and depth of the voxel grid. In at least one embodiment, the voxel grid contains all points of the point cloud. In at least one embodiment, voxel grid resolution refers to the number of voxels that fit within given dimensions (height, width, depth) of the voxel grid. In at least one embodiment, the array of voxel grid resolution information is identified in pseudocode having a name such as voxel_size.

[0039] In at least one embodiment, operation 401 includes a processor that allocates memory locations to store an array (or other tensor) of point cloud data. In at least one embodiment, the point cloud data includes the number of points, the coordinates of each point, the features of each point, or a combination of some of these. In at least one embodiment, the array of point cloud data is identified in pseudocode having a name such as point_cloud. In at least one embodiment, the processor allocates one or more memory locations, such as a buffer, to store various aspects of the point cloud data, such as the coordinates of points, the features of each point, or a combination of some of these.

[0040] In at least one embodiment, operation 401 includes a processor that allocates memory locations to store an array of voxel IDs (or other tensor) and / or average feature values, which is further described herein together with at least FIG. 3. In at least one embodiment, the array of voxel IDs and / or average feature values ​​is identified in pseudocode having a name such as output_space.

[0041] In at least one embodiment, operation 401 includes a processor that allocates memory locations to store data of a voxelized hash table, which is further described herein together with at least FIGS. 1 to 3 and FIGS. 5 to 6. In at least one embodiment, the amount of memory locations allocated to the voxelized hash table is at least twice the number of points in the point cloud. In at least one embodiment, arrays used to store data of the voxelized hash table are identified in pseudocode having a name such as hash_table. In at least one embodiment, the array of the voxelized hash table stores voxel offset values ​​in an array identified in pseudocode having a name such as voxel_offset.

[0042] In at least one embodiment, when memory locations are allocated from data storage devices in operation 401, the processor continues the process (400) in operation 402 by performing operations that cause the processor to calculate one or more voxel offsets, which is further described herein with at least FIG. 2. In at least one embodiment, the processor performs the operation of one or more functions to map each point of a point cloud to a voxel in a voxel grid, as further described herein with at least FIG. 1. In at least one embodiment, the function for mapping each point of a point cloud to a voxel in a voxel grid is identified in pseudocode having a name such as map_point_to_voxel(). In at least one embodiment, the processor uses a function such as map_point_to_voxel() to calculate the voxel offset of each point based on the 3D coordinates of each point and information regarding the resolution of the voxel grid stored in voxel_size. In at least one embodiment, the processor stores the voxel offset values ​​in an array such as voxel_offset. In at least one embodiment, operation 402 includes a processor that performs operations to calculate the voxel index value of each voxel offset, which is further described in this specification together with at least FIG. 2.

[0043] In at least one embodiment, the processor continues the process (400) to operation 404 by performing operations that cause the processor to load / store voxel offsets and voxel index values ​​into a voxelized hash table, which is further described herein together with at least FIG. 2. In at least one embodiment, the processor loads / stores entries corresponding to each point in the point cloud in the voxelized hash table by pairing keys and values ​​(key, value) using arrays voxel_offset and voxel_idx (voxel_offset, voxel_idx). In at least one embodiment, the voxelized hash table is used by the processor to correlate a voxel ID with a point in the point cloud using voxel offset and voxel index values, and an indication of the memory location where the corresponding voxel ID is stored or to be stored.

[0044] In at least one embodiment, the processor continues the process (400) to operation 406 by performing operations that cause the processor to load / store the sum of the point feature values ​​of each voxel and the number of points in each voxel in an array such as output_space. In at least one embodiment, the processor traverses one or more point locations stored in a voxelization hash table and partially generates the sum of the feature values ​​in each voxel. In at least one embodiment, the processor traverses the point locations in the voxelization hash table to identify the point locations and thus identify points common to each voxel ID stored in the voxelization hash table. In at least one embodiment, the processor traverses the point locations in the voxelization hash table to identify data values ​​such as the point feature values ​​of the points corresponding to those point locations and generates the sum of the feature values ​​of each voxel. In at least one embodiment, the processor stores one or more data values ​​representing features associated with one or more point locations in one or more memory locations, such as a buffer, which are to be accessed at least partially based on data stored in one or more data structures such as hash tables. In at least one embodiment, point feature values ​​of points identified as being associated with a common voxel ID are loaded / stored in an array such as output_space, then summed together by the voxel ID, and loaded / stored in output_space. In at least one embodiment, feature values ​​of different types are weighted differently before being summed by the processor.In at least one embodiment, the processor calculates the sum of all feature values ​​in each voxel based at least partially on iterative calculations of voxel offsets for each point in the point cloud to store an ordered array or queue of voxel offsets that look up corresponding voxel IDs in order using a voxelization hash table. In at least one embodiment, the processor looks up voxel IDs corresponding to different voxel offsets in parallel. In at least one embodiment, for each voxel ID, the processor identifies individual points in the point cloud at that voxel ID using the corresponding voxel offset. In at least one embodiment, when the processor identifies individual points within each voxel ID, the processor may identify the feature values ​​of those individual points by accessing point cloud data and / or point feature values ​​stored in a buffer, and sum those point feature values ​​to be stored in an array such as output_space. In at least one embodiment, also in operation 406, the processor calculates the number of points per voxel ID based on the identification of the individual points mapped to each voxel ID. In at least one embodiment, the number of points mapped to each voxel ID is calculated using atomic operations and / or functions, such as those represented by pseudocode having atomicADD, atomicINC, etc.

[0045] In at least one embodiment, the processor continues process (400) to operation 408 by using values ​​loaded / stored in an array such as output_space to calculate an average feature value for each voxel. In at least one embodiment, the processor calculates an average feature value for a voxel by using the sum of the feature values ​​for the voxel and dividing it by the number of points in the voxel, which is further described herein together with at least FIG. 3. In at least one embodiment, the average feature value is used by the processor to generate, display, or otherwise activate a voxel representing the average feature value on a display device.

[0046] FIG. 5 illustrates a block diagram of a process (500) that uses a processor to perform an API function that enables one or more processors to perform voxelization operations using a voxelization hash table and as otherwise described herein, according to at least one embodiment. In at least one embodiment, one or more aspects of one or more embodiments described herein together with FIG. 5 are combined with one or more aspects of one or more embodiments described herein, including those described together with FIG. 1 through 4 and FIG. 6. In at least one embodiment, one or more processors perform one or more operations of the process (500). In at least one embodiment, one or more processors performing one or more operations of the process (500) are any one processor or combination of processors described herein, including the processor(s) (102) described with FIG. 1, the CPU (1302) described with FIG. 13, the accelerator(s) (1014) described with FIG. 10c, the graphics processor (1710) described with FIG. 17a, and the parallel processing unit ("PPU") (3200) described with FIG. 32. In at least one embodiment, the processor(s) (102) perform one or more operations of the process (500), such as performing API voxelization functions using a voxelization hash table as operation 504. In at least one embodiment, one or more operations of the process (500) are one or more operations of the system (200) of FIG. 2, such as inputting point cloud data or their representations into the API(s) as operation 502. In at least one embodiment, one or more operations of the process (500) are combined with one or more operations of the system (300) of FIG. 3, such as those of the voxelization module (304). In at least one embodiment, one or more operations of the process (500) are one or more operations of the process (400), such as calculating the voxel offsets of operation 404.In at least one embodiment, one or more operations of the process (500) are one or more operations of the API(s) (610), such as performing the API voxelization function(s) of operation 504.

[0047] In at least one embodiment, a user (or application) initiates a process (500) in operation 502 by inputting point cloud data or its representations into an API function. In at least one embodiment, the API function is further described herein with at least Fig. 6. In at least one embodiment, the API function is part of an API library used to perform AI operations, such as NVIDIA® CUDA® and AMD® ROCm®. In at least one embodiment, the user interface is a mobile application, a website, a customer portal, an AI-assisted conversational user interface, or a combination thereof. In at least one embodiment, the user causes the processor to automatically input point cloud data into the API function, such as streaming point cloud data based on sensors installed on an autonomous vehicle, such as the autonomous vehicle (1000) described with Fig. 10a.

[0048] In at least one embodiment, the processor continues the process (500) to operation 504 by causing the execution of one or more API voxelization function(s) using a voxelization hash table. In at least one embodiment, the API voxelization function is a hash function. In at least one embodiment, the API voxelization function allocates memory locations to one or more data storage locations as further described herein with at least FIG. 4. In at least one embodiment, the API voxelization function is map_point_to_voxel() as further described herein with at least FIG. 4. In at least one embodiment, the API voxelization function inserts entries into the voxelization hash table as further described herein with at least FIG. 4. In at least one embodiment, the API voxelization function sums all feature points and calculates the total number of points in each voxel as further described herein with at least FIG. 4. In at least one embodiment, the API voxelization function calculates the average feature of each valid voxel as further described in this specification together with at least FIG. 4.

[0049] In at least one embodiment, the processor continues the process (500) to operation 506 by outputting a voxel representation of the environment based on point cloud data. In at least one embodiment, the processor performs operations to generate, display, or activate one or more valid voxels on a display device using the average features of each valid voxel, as otherwise described in this specification together with at least FIG. 2.

[0050] FIG. 6 illustrates a block diagram of a driver and / or runtime comprising one or more libraries for providing one or more application programming interfaces (APIs) according to at least one embodiment. In at least one embodiment, any one processor, or a combination of processors, performs API(s) (610), comprising the processor(s) (102) of FIG. 1, the CPU (1302) described with FIG. 13, the graphics processor (1710) described with FIG. 17a, and the parallel processing unit ("PPU") (3200) described with FIG. 32. In at least one embodiment, the API(s) (610) are further described herein. In at least one embodiment, a call to the API(s) (610) causes any one or more operations of any one or more modules of FIG. 1 through 3 to be performed. In at least one embodiment, a call to the API(s) (610) causes the processor to perform any one or more operations described with FIG. 1 through 5 to be performed. In at least one embodiment, the API(s) (610) receive point cloud data, or a representation thereof, as input and cause the voxelization module (104) of FIG. 1 to perform operations to voxelize a point cloud representation of the environment, in part, using a voxelization hash table. In at least one embodiment, the API(s) (610) receive point cloud data, or a representation thereof and cause the voxelization module (204) to perform one or more operations to load / store voxel offsets into a voxelization hash table. In at least one embodiment, a call to the API(s) (610) causes the processor to perform one or more operations of the voxelization module (304), such as calculating a feature average for each voxel in a voxel grid. In at least one embodiment, one or more operations to be performed by the processor when the API(s) (610) is called are described in the process (400) of FIG. 4 and the process (500) of FIG. 5.

[0051] In at least one embodiment, the software program (602) is a software module. In at least one embodiment, the software program (602) includes one or more software modules. In at least one embodiment, one or more APIs (610) are sets of software instructions that, when executed, cause one or more processors to perform one or more computing operations. In at least one embodiment, one or more APIs (610) are distributed or otherwise provided as part of one or more libraries (606), runtimes (604), drivers (604), and / or any other grouping of software and / or executable code further described herein. In at least one embodiment, one or more APIs (610) perform one or more computing operations in response to a call by the software programs (602). In at least one embodiment, the software program (602) is a collection of software code, commands, instructions, or other sequences of text for instructing a computing device to perform one or more computing operations and / or to call one or more other sets of instructions, such as APIs (610) or function(s) (612), to be executed. In at least one embodiment, the functionality provided by one or more APIs (610) includes software functions that can be used to accelerate one or more parts of software programs (602) using one or more parallel processing units (PPUs), such as graphics processing units (GPUs). In at least one embodiment, the software program is a compiler.

[0052] In at least one embodiment, the APIs (610) are hardware interfaces to one or more circuits for performing one or more computing operations. In at least one embodiment, one or more software APIs (610) described herein are implemented as one or more circuits for performing one or more techniques described herein. In at least one embodiment, one or more software programs (602) include instructions that, when executed, cause one or more hardware devices and / or circuits to perform one or more techniques further described herein.

[0053] In at least one embodiment, software programs (602), such as user-implemented software programs, utilize one or more application programming interfaces (APIs) (610) to perform various computing operations, such as memory reservation, matrix multiplication, arithmetic operations, or any computing operations performed by parallel processing units (PPUs), such as graphics processing units (GPUs), as further described herein. In at least one embodiment, one or more APIs (610) provide a set of callable function(s) (612), referred to herein as APIs, API functions, and / or functions, which individually perform one or more computing operations, such as computing operations related to parallel computing. For example, in an embodiment, one or more APIs (610) provide function(s) (612) that cause a scheduler to schedule instructions to be executed by processors based on the latency of interconnects coupled to these processors. In at least one embodiment, the API(s) (610) provides one or more function(s) (612) which are one or more neural networks, such as a neural network trained to classify objects in images and implemented on the neural network classification module (106) of FIG. 1.

[0054] In at least one embodiment, one or more software programs (602) interact with or otherwise communicate with one or more APIs (610) to perform one or more computing operations using one or more PPUs, such as GPUs. In at least one embodiment, the one or more computing operations using the one or more PPUs include at least one or more groups of computing operations that are at least partially accelerated by execution by the one or more PPUs. In at least one embodiment, one or more software programs (602) interact with one or more APIs (610) using a remote or local interface to facilitate parallel computing.

[0055] In at least one embodiment, the interface is a set of software instructions that, when executed, provide access to one or more function(s) (612) provided by one or more APIs (610). In at least one embodiment, the software program (602) uses a local interface when a software developer compiles the software program (602) with one or more libraries (606) that include or otherwise provide access to one or more APIs (610). In at least one embodiment, the software program (602) is statically compiled with uncompiled source code or pre-compiled libraries (606) that include instructions for executing one or more APIs (610). In at least one embodiment, the software program (602) is dynamically compiled, and the software program utilizes a linker to link to one or more pre-compiled libraries (606) that include one or more APIs (610).

[0056] In at least one embodiment, the software program (602) uses a remote interface when a software developer executes a software program that uses a library (606) containing one or more APIs (610) or otherwise communicates with it through a network or other remote communication medium. In at least one embodiment, one or more libraries (606) containing one or more APIs (610) will be executed by a remote computing service, such as a computing resource service provider. In another embodiment, one or more libraries (606) containing one or more APIs (610) will be executed by any other computing host that provides said one or more APIs (610) to one or more software programs (602).

[0057] In at least one embodiment, a processor executing or using one or more software programs (602) calls, uses, executes, or otherwise implements one or more APIs (610) to allocate and otherwise manage memory to be used by said software programs (602). In at least one embodiment, one or more software programs (602) utilize one or more APIs (610) to allocate and otherwise manage memory to be used by one or more parts of said software programs (602) to be accelerated using one or more PPUs, such as GPUs or any other accelerator or processor further described herein. In an embodiment, said software programs (602) may be executed by one or more processors based, at least partially, on the latency of interconnects coupled to one or more processors using function(s) (612) provided by one or more APIs (610).

[0058] In at least one embodiment, the API (610) is an API for facilitating parallel computing. In at least one embodiment, the API (610) is any other API further described herein. In at least one embodiment, the API (610) is provided by a driver and / or runtime (604). In at least one embodiment, the API (610) is provided by a CUDA user-mode driver. In at least one embodiment, the API (610) is provided by a CUDA runtime. In at least one embodiment, the driver (604) is data values ​​and software instructions that, when executed, perform or otherwise facilitate the operation of one or more function(s) (612) of the API (610) during the loading and execution of one or more parts of a software program (602). In at least one embodiment, the runtime (604) is data values ​​and software instructions that, when executed, perform or otherwise facilitate the operation of one or more function(s) (612) of the API (610) during the execution of a software program (602). In at least one embodiment, one or more software programs (602) utilize one or more APIs (610) implemented or otherwise provided by a driver and / or runtime (604) to perform arithmetic operations combined by said one or more software programs (602) during execution by said one or more PPUs, such as GPUs.

[0059] In at least one embodiment, one or more software programs (602) utilize one or more APIs (610) provided by a driver and / or runtime (604) to perform combined arithmetic operations of one or more PPUs, such as GPUs. In at least one embodiment, one or more APIs (610) provide combined arithmetic operations through the driver and / or runtime (604) as described above. In at least one embodiment, one or more software programs (602) utilize one or more APIs (610) provided by a driver and / or runtime (604) to allocate or otherwise reserve one or more blocks of memory (614) of one or more PPUs, such as GPUs. In at least one embodiment, one or more software programs (602) utilize one or more APIs (610) provided by a driver and / or runtime (604) to allocate or otherwise reserve blocks of memory. In at least one embodiment, one or more APIs (610) perform combined mathematical functions as described herein.

[0060] In at least one embodiment, to improve the usability of a software program (602) and / or the optimization of one or more parts of said software programs (602) to be accelerated by one or more PPUs such as GPUs, one or more APIs (610) provide one or more API function(s) (612) for performing a scheduling system available or used by one or more computing devices as described herein. In at least one embodiment, a processor performs one or more software programs to combine two or more application programming interfaces (APIs) into a single API. In at least one embodiment, a processor uses APIs to cause a scheduler to select a thread selection mechanism and / or to perform the operations described herein in a different way. In at least one embodiment, APIs call a scheduler to cause resource allocation. In at least one embodiment, a processor uses exemplary APIs to schedule one or more instructions to be performed by one or more processors based, at least partially, on the latency of one or more interconnects coupled to these one or more processors.

[0061] In at least one embodiment, memory (614) is system memory (1704) of a computing system (1700). In at least one embodiment, memory (614) is any form of hardware that stores data and is referred to as storage or data storage. In at least one embodiment, memory (614) stores data of any one or more arrays described together with FIG. 4. In at least one embodiment, memory (614) stores data of a voxelized hash table as described together with FIG. 2 and as otherwise described herein. In at least one embodiment, memory (614) stores the weights of the neural network of the neural network classification module (106) of FIG. 1, the voxel IDs (206a-) of FIG. 2n ), voxel of Fig. 3 (314a- n It stores data used in various operations described in this specification, including average feature values ​​per operation, voxel index values ​​of operation 404 of FIG. 4, and point cloud data input to the API(s) as operation 502 of FIG. 5.

[0062] In at least one embodiment, memory (614) is a computer-readable storage medium and / or code stored on the computer-readable storage medium in the form of a computer program comprising a plurality of computer-readable instructions executable by one or more processors. In at least one embodiment, the computer-readable storage medium is a non-transient computer-readable medium. In at least one embodiment, at least some computer-readable instructions available to perform operations described in connection with FIG. 1 are not stored using only transient signals (e.g., propagating transient electrical or electromagnetic transmissions). In at least one embodiment, the non-transient computer-readable medium does not necessarily include non-transient data storage circuits (e.g., buffers, caches, and queues) within transient signal transmitters and receivers. In at least one embodiment, memory (614) is implemented as a non-transient computer-readable storage medium storing executable instructions that, when executed by one or more processors of a computer system, cause the computer system to infer computer system architecture designs as further described herein together with FIGS. 1 through 6.

[0063] logic

[0064] FIG. 7a illustrates logic (715) that may be used in one or more devices to perform operations such as those discussed herein according to at least one embodiment, as described elsewhere in this specification. In at least one embodiment, the logic (715) is used to perform inference and / or training operations associated with one or more embodiments. In at least one embodiment, the logic (715) is inference and / or training logic. Details regarding the logic (715) are provided below together with FIG. 7a and / or FIG. 7b. In at least one embodiment, the logic refers to any combination of software logic, hardware logic, and / or firmware logic to provide the functionality or operations described herein, wherein the logic may be implemented as a circuit forming part of a larger system, e.g., an integrated circuit (IC), a system-on-chip (SoC), or one or more processors (e.g., CPU, GPU), either collectively or individually.

[0065] In at least one embodiment, the logic (715) may include, without limitation, code and / or data storage (701) for storing forward and / or output weights and / or input / output data and / or other parameters for configuring neurons or layers of a neural network that are trained and / or used for inference in one or more embodiments. In at least one embodiment, the logic (715) may include or be combined with code and / or data storage (701) for storing graph code or other software that controls the timing and / or order in which weight and / or other parameter information must be loaded to configure logic that includes integer and / or floating-point units (collectively, arithmetic logic units (ALUs)). In at least one embodiment, code such as graph code loads weight or other parameter information to processor ALUs based on the architecture of the neural network that corresponds to such code. In at least one embodiment, the code and / or data storage (701) stores the weight parameters and / or input / output data of each layer of a neural network that is trained or used with one or more embodiments during the forward propagation of the input / output data and / or weight parameters during training and / or inference using an aspect of one or more embodiments. In at least one embodiment, any part of the code and / or data storage (701) may be included with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory.

[0066] In at least one embodiment, any part of the code and / or data storage (701) may be inside or outside of one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or code and / or data storage (701) may be cache memory, "DRAM" (dynamic randomly addressable memory), "SRAM" (static randomly addressable memory), non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the choice of whether the code and / or code and / or data storage (701) is, for example, inside or outside of a processor, or includes DRAM, SRAM, flash, or some other storage type may depend on available storage on-chip versus off-chip, latency requirements of the training and / or inference functions being performed, the batch size of the data used for inference and / or training of the neural network, or some combination of these factors.

[0067] In at least one embodiment, the logic (715) may include, without limitation, a code and / or data storage (705) that stores reverse and / or output weights and / or input / output data corresponding to neurons or layers of a neural network trained and / or used for inference in one or more embodiments. In at least one embodiment, the code and / or data storage (705) stores weight parameters and / or input / output data of each layer of a neural network trained or used with one or more embodiments during the reverse propagation of input / output data and / or weight parameters during training and / or inference using one or more embodiments. In at least one embodiment, the logic (715) may include or be coupled thereto a code and / or data storage (705) for storing graph code or other software that controls the timing and / or order in which weight and / or other parameter information must be loaded to construct logic comprising integer and / or floating-point units (collectively, arithmetic logic units (ALUs)).

[0068] In at least one embodiment, a code such as a graph code causes the loading of weight or other parameter information into processor ALUs based on the architecture of the neural network corresponding to such code. In at least one embodiment, any part of the code and / or data storage (705) may be included together with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory. In at least one embodiment, any part of the code and / or data storage (705) may be inside or outside of one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data storage (705) may be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the choice of whether the code and / or data storage (705) is, for example, inside or outside the processor, or includes DRAM, SRAM, flash memory, or some other storage type may depend on the available storage on-chip versus off-chip, the latency requirements of the training and / or inference functions being performed, the batch size of the data used for the inference and / or training of the neural network, or some combination of these factors.

[0069] In at least one embodiment, the code and / or data storage (701) and the code and / or data storage (705) may be separate storage structures. In at least one embodiment, the code and / or data storage (701) and the code and / or data storage (705) may be combined storage structures. In at least one embodiment, the code and / or data storage (701) and the code and / or data storage (705) may be partially combined and partially separated. In at least one embodiment, any part of the code and / or data storage (701) and the code and / or data storage (705) may be included together with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory.

[0070] In at least one embodiment, the logic (715) may, without limitation, include one or more "ALU(s)" (arithmetic logic unit(s)) (710) comprising integer and / or floating-point units that perform logical and / or mathematical operations based at least partially on, or represented by, a training and / or inference code (e.g., graph code), and the result may produce activations (e.g., output values ​​from layers or neurons in a neural network) stored in an activation storage (720) which are functions of input / output and / or weight parameter data stored in the code and / or data storage (701) and / or the code and / or data storage (705). In at least one embodiment, activations stored in the activation storage (720) are generated according to linear algebra and / or matrix-based mathematics performed by the ALU(s) (710) in response to the execution of instructions or other code, wherein weight values ​​stored in the code and / or data storage (705) and / or data storage (701) are used as operands together with other values ​​such as bias values, gradient information, momentum values, or other parameters or hyperparameters, any or all of which may be stored in the code and / or data storage (705) or the code and / or data storage (701) or other storage on or off-chip.

[0071] In at least one embodiment, the ALU(s) (710) are contained within one or more processors or other hardware logic devices or circuits, whereas in other embodiments, the ALU(s) (710) may be outside the processor or other hardware logic devices or circuits using them (e.g., coprocessors). In at least one embodiment, the ALUs (710) may be contained within the execution units of the processor or in a bank of ALUs accessible by the execution units of the processor distributed within the same processor or among different types of different processors (e.g., central processing units, graphics processing units, fixed function units, etc.). In at least one embodiment, the code and / or data storage (701), code and / or data storage (705), and activation storage (720) may share the processor or other hardware logic devices or circuits, whereas in other embodiments, they may be in different processors or other hardware logic devices or circuits, or in some combination of the same and different processors or other hardware logic devices or circuits. In at least one embodiment, any part of the activation storage (720) may be included in other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory. Additionally, the inference and / or training code may be stored together with other code accessible to the processor or other hardware logic or circuits, and may be fetched and / or processed using the processor's fetch, decoding, scheduling, execution, retirement and / or other logic circuits.

[0072] In at least one embodiment, the activation storage (720) may be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the activation storage (720) may be wholly or partially located within or outside of one or more processors or other logic circuits. In at least one embodiment, the choice of whether the activation storage (720) is, for example, inside or outside of a processor, or includes DRAM, SRAM, flash memory, or some other storage type may depend on the available storage on-chip versus off-chip, the latency requirements of the training and / or inference functions being performed, the batch size of the data used for the inference and / or training of the neural network, or some combination of these factors.

[0073] In at least one embodiment, the logic (715) illustrated in FIG. 7a may be used with an application-specific integrated circuit ("ASIC"), such as a TensorFlow® processing unit from Google, an inference processing unit (IPU) from Graphcore™, or a Nervana® (e.g., "Lake Crest") processor from Intel Corp. In at least one embodiment, the logic (715) illustrated in FIG. 7a may be used with other hardware, such as central processing unit ("CPU") hardware, graphics processing unit ("GPU") hardware, or field programmable gate arrays ("FPGAs").

[0074] FIG. 7b illustrates logic (715) according to at least one embodiment. In at least one embodiment, the logic (715) is inference and / or training logic. In at least one embodiment, the logic (715) may include, without limitation, hardware logic in which computing resources are dedicated or otherwise exclusively used with weight values ​​or other information corresponding to one or more layers of neurons in a neural network. In at least one embodiment, the logic (715) illustrated in FIG. 7b may be used with an application-specific integrated circuit (ASIC), such as a TensorFlow® processing unit from Google, an inference processing unit (IPU) from Graphcore™, or a Nervana® (e.g., "Lake Crest") processor from Intel Corp. In at least one embodiment, the logic (715) illustrated in FIG. 7b may be used with other hardware, such as a central processing unit (CPU) hardware, a graphics processing unit (GPU) hardware, or field programmable gate arrays (FPGAs). In at least one embodiment, the logic (715) includes, without limitation, a code and / or data storage (701) and a code and / or data storage (705), which may be used to store other information including a code (e.g., graph code), weight values, and / or bias values, gradient information, momentum values, and / or other parameters or hyperparameter information. In at least one embodiment illustrated in FIG. 7b, the code and / or data storage (701) and the code and / or data storage (705) are each associated with a dedicated computing resource, such as computing hardware (702) and computing hardware (706).In at least one embodiment, each of the computing hardware (702) and the computing hardware (706) includes one or more ALUs that perform mathematical functions, such as linear algebraic functions, only for information stored in the code and / or data storage (701) and the code and / or data storage (705), respectively, and the result is stored in the activation storage (720).

[0075] In at least one embodiment, the code and / or data storage (701 and 705) and the corresponding computing hardware (702 and 706) each correspond to different layers of the neural network, and the resulting activation from one storage / computing pair (701 / 702) of the code and / or data storage (701) and computing hardware (702) is provided as an input to the next storage / computing pair (705 / 706) of the code and / or data storage (705) and computing hardware (706) to mirror the conceptual configuration of the neural network. In at least one embodiment, each of the storage / computing pairs (701 / 702 and 705 / 706) may correspond to more than one layer of the neural network. In at least one embodiment, additional storage / computing pairs (not shown) following or parallel to the storage / computing pairs (701 / 702 and 705 / 706) may be included in the logic (715).

[0076] Neural network training and deployment

[0077] FIG. 8 illustrates the training and deployment of a deep neural network according to at least one embodiment. In at least one embodiment, an untrained neural network (806) is trained using a training dataset (802). In at least one embodiment, the training framework (804) is the PyTorch framework, whereas in other embodiments, the training framework (804) is TensorFlow, Boost, Caffe, Microsoft Cognitive Toolkit / CNTK, MXNet, Chainer, Keras, Deeplearning4j, or another training framework. In at least one embodiment, the training framework (804) enables the untrained neural network (806) to be trained using the processing resources described herein to generate a trained neural network (808). In at least one embodiment, weights may be selected by pre-training using a deep belief network or randomly. In at least one embodiment, training may be performed in a supervised, partially supervised, or unsupervised manner.

[0078] In at least one embodiment, the untrained neural network (806) is trained using supervised learning, wherein the training dataset (802) includes inputs paired with desired outputs for the inputs, or the training dataset (802) includes inputs having known outputs and the outputs of the neural network (806) are manually graded. In at least one embodiment, the untrained neural network (806) is trained in a supervised manner, processes inputs from the training dataset (802), and compares the resulting outputs against a set of expected or desired outputs. In at least one embodiment, errors are then propagated back through the untrained neural network (806). In at least one embodiment, the training framework (804) adjusts the weights controlling the untrained neural network (806). In at least one embodiment, the training framework (804) includes tools for monitoring how well an untrained neural network (806) is converging toward a model, such as a trained neural network (808), which is suitable for generating accurate answers, as in the result (814), based on input data such as a new dataset (812). In at least one embodiment, the training framework (804) iteratively trains the untrained neural network (806) by adjusting weights to refine the output of the untrained neural network (806) using a loss function and adjustment algorithm, such as stochastic gradient descent. In at least one embodiment, the training framework (804) trains the untrained neural network (806) until the untrained neural network (806) achieves a desired accuracy. In at least one embodiment, the trained neural network (808) may then be deployed to implement any number of machine learning operations.

[0079] In at least one embodiment, an untrained neural network (806) is trained using unsupervised learning, wherein the untrained neural network (806) attempts to train itself using unlabeled data. In at least one embodiment, the unsupervised learning training dataset (802) will include input data without any associated output data or "real-world data" data. In at least one embodiment, the untrained neural network (806) can learn groupings within the training dataset (802) and determine how individual inputs relate to the untrained dataset (802). In at least one embodiment, unsupervised training may be used to generate a self-organizing map in a trained neural network (808) capable of performing operations useful for reducing the dimensionality of the new dataset (812). In at least one embodiment, unsupervised training may also be used to perform anomaly detection, which allows for the identification of data points in the new dataset (812) that deviate from the normal patterns of the new dataset (812).

[0080] In at least one embodiment, semi-supervised learning may be used, wherein the training dataset (802) is a technique comprising a mixture of labeled data and unlabeled data. In at least one embodiment, the training framework (804) may be used to perform incremental learning, as with the transmitted learning techniques. In at least one embodiment, incremental learning enables the trained neural network (808) to adapt to a new dataset (812) without forgetting the knowledge injected into the trained neural network (808) during the initial training.

[0081] In at least one embodiment, the training framework (804) is a framework processed in relation to a software development toolkit, such as the OpenVINO (Open Visual Inference and Neural Network Optimization) toolkit. In at least one embodiment, the OpenVINO toolkit is a toolkit such as those developed by Intel Corporation in Santa Clara, California. In at least one embodiment, OpenVINO includes logic (715) or uses logic (715) to perform the operations described herein. In at least one embodiment, an SoC, integrated circuit, or processor uses OpenVINO to perform the operations described herein.

[0082] In at least one embodiment, OpenVINO is a toolkit for facilitating the development of applications for various tasks and operations, specifically neural network applications, such as human vision emulation, speech recognition, natural language processing, recommendation systems, and / or variations thereof. In at least one embodiment, OpenVINO supports neural networks such as convolutional neural networks (CNNs), recurrent and / or attention-based neural networks, and / or various other neural network models. In at least one embodiment, OpenVINO supports various software libraries such as OpenCV, OpenCL, and / or variations thereof.

[0083] In at least one embodiment, OpenVINO supports neural network models for various tasks and actions, such as classification, segmentation, object detection, face recognition, speech recognition, pose estimation (e.g., humans and / or objects), monocular depth estimation, image inpainting, style transfer, action recognition, coloring, and / or variations thereof.

[0084] In at least one embodiment, OpenVINO includes one or more software tools and / or modules for model optimization, also referred to as a model optimizer. In at least one embodiment, the model optimizer is a command-line tool that facilitates transitions between training and deployment of neural network models. In at least one embodiment, the model optimizer optimizes neural network models for execution on various devices and / or processing units, such as GPUs, CPUs, PPUs, GPGPUs, and / or variations thereof. In at least one embodiment, the model optimizer generates an internal representation of the model and optimizes the model to generate an intermediate representation. In at least one embodiment, the model optimizer reduces the number of layers in the model. In at least one embodiment, the model optimizer removes layers of the model used for training. In at least one embodiment, the model optimizer performs various neural network operations, such as modifying inputs to the model (e.g., resizing inputs to the model), modifying the size of inputs to the model (e.g., modifying the batch size of the model), modifying the structure of the model (e.g., modifying the layers of the model), normalization, standardization, quantization (e.g., converting weights of the model from a first representation such as floating-point to a second representation such as integers), and / or variations thereof.

[0085] In at least one embodiment, OpenVINO includes one or more software libraries for inference, also referred to as an inference engine. In at least one embodiment, the inference engine is a C++ library or any suitable programming language library. In at least one embodiment, the inference engine is utilized to infer input data. In at least one embodiment, the inference engine implements various classes to infer input data and generate one or more results. In at least one embodiment, the inference engine implements one or more API functions to process intermediate representations, set input and / or output formats, and / or run a model on one or more devices.

[0086] In at least one embodiment, OpenVINO provides various capabilities for heterogeneous execution of one or more neural network models. In at least one embodiment, heterogeneous execution, or heterogeneous computing, refers to one or more computing processes and / or systems utilizing one or more types of processors and / or cores. In at least one embodiment, OpenVINO provides various software functions for executing a program on one or more devices. In at least one embodiment, OpenVINO provides various software functions for executing a program and / or parts of a program on different devices. In at least one embodiment, OpenVINO provides various software functions for executing, for example, a first part of code on a CPU and a second part of code on a GPU and / or FPGA. In at least one embodiment, OpenVINO provides various software functions for executing one or more layers of a neural network on one or more devices (for example, a first set of layers on a first device such as a GPU, and a second set of layers on a second device such as a CPU).

[0087] In at least one embodiment, OpenVINO includes various functionalities similar to functionalities associated with CUDA programming models, such as various neural network model operations associated with frameworks such as TensorFlow, PyTorch, and / or variations thereof. In at least one embodiment, one or more CUDA programming model operations are performed using OpenVINO. In at least one embodiment, the various systems, methods, and / or techniques described herein are implemented using OpenVINO.

[0088] data center

[0089] FIG. 9 illustrates an exemplary data center (900) in which at least one embodiment may be used. In at least one embodiment, the data center (900) includes a data center infrastructure layer (910), a framework layer (920), a software layer (930), and an application layer (940).

[0090] In at least one embodiment, as illustrated in FIG. 9, the data center infrastructure layer (910) may include a resource orchestrator (912), grouped computing resources (914), and node computing resources ("node CRs") (916(1)-916(N)), where "N" represents a positive integer (which may be an integer "N" different from that used in other drawings). In at least one embodiment, the node CRs (916(1)-916(N)) may include, but are not limited to, any number of "CPUs" (central processing units) or other processors (including accelerators, FPGAs (field programmable gate arrays), graphics processors, etc.), memory storage devices (918(1)-918(N)) (e.g., dynamic read-only memory, solid-state storage, or disk drives), "NW I / O (network input / output) devices, network switches, "VMs" (virtual machines), power modules, and cooling modules, etc. In at least one embodiment, one or more node CRs from the node CRs (916(1)-916(N)) may be servers having one or more of the aforementioned computing resources.

[0091] In at least one embodiment, the grouped computing resources (914) may include node CRs housed within one or more racks (not shown), or individual groups of many racks (also not shown) housed in data centers at various geographic locations. In at least one embodiment, the individual groups of node CRs within the grouped computing resources (914) may include grouped computing, network, memory, or storage resources that may be configured or allocated to support one or more workloads. In at least one embodiment, multiple node CRs including CPUs or processors may be grouped within one or more racks to provide computing resources to support one or more workloads. In at least one embodiment, one or more racks may also include any number of power modules, cooling modules, and network switches in any combination.

[0092] In at least one embodiment, the resource orchestrator (912) may organize or otherwise control one or more node CRs (916(1)-916(N)) and / or grouped computing resources (914). In at least one embodiment, the resource orchestrator (912) may include a "software design infrastructure" (SDI) management entity for the data center (900). In at least one embodiment, the resource orchestrator (712) may include hardware, software, or a combination of some of these.

[0093] In at least one embodiment, as illustrated in FIG. 9, the framework layer (920) includes a task scheduler (922), a configuration manager (924), a resource manager (926), and a distributed file system (928). In at least one embodiment, the framework layer (920) may include a framework that supports software (932) of the software layer (930) and / or one or more application(s) (942) of the application layer (940). In at least one embodiment, the software (932) or application(s) (942) may each include web-based service software or applications such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. In at least one embodiment, the framework layer (920) may be a type of free and open-source software web application framework, such as Apache Spark™ (hereinafter “Spark”), which can utilize the distributed file system (928) for large-scale data processing (e.g., “big data”), but is not limited thereto. In at least one embodiment, the job scheduler (922) may include a Spark driver that facilitates the scheduling of workloads supported by various layers of the data center (900). In at least one embodiment, the configuration manager (924) may configure different layers, such as a framework layer (920) and a software layer (930), which include Spark and a distributed file system (928) to support large-scale data processing. In at least one embodiment, the resource manager (926) may manage clustered or grouped computing resources allocated or mapped thereto for the support of the distributed file system (928) and the job scheduler (922).In at least one embodiment, clustered or grouped computing resources may include computing resources (914) grouped in a data center infrastructure layer (910). In at least one embodiment, a resource manager (926) may coordinate with a resource orchestrator (912) to manage these mapped or allocated computing resources.

[0094] In at least one embodiment, the software (932) included in the software layer (930) may include software used by at least some of the node CRs (916(1)-916(N)), grouped computing resources (914), and / or the distributed file system (928) of the framework layer (920). In at least one embodiment, one or more types of software may include, but are not limited to, Internet web page search software, email virus scan software, database software, and streaming video content software.

[0095] In at least one embodiment, the application(s) (942) included in the application layer (940) may include at least some of the node CRs (916(1)-916(N)), grouped computing resources (914), and / or one or more types of applications used by the distributed file system (928) of the framework layer (920). In at least one embodiment, one or more types of applications may include any number of genomics applications, cognitive computing applications, and machine learning applications, including, but not limited to, training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), or other machine learning applications used with one or more embodiments.

[0096] In at least one embodiment, any of the configuration manager (924), resource manager (926), and resource orchestrator (912) may implement any number and type of self-correcting actions based on any amount and type of data acquired in any technically feasible manner. In at least one embodiment, the self-correcting actions may mitigate the data center operator of the data center (900) from making bad configuration decisions and avoiding underutilized and / or poorly performing parts of the data center.

[0097] In at least one embodiment, the data center (900) may include tools, services, software, or other resources for training one or more machine learning models according to one or more embodiments described herein, or for using one or more machine learning models to predict or infer information. For example, in at least one embodiment, the machine learning model may be trained by calculating weight parameters according to a neural network architecture using the software and computing resources described above with respect to the data center (900). In at least one embodiment, the trained machine learning model corresponding to one or more neural networks may be used to infer or predict information using the resources described above with respect to the data center (900) by using weight parameters calculated through one or more training techniques described herein.

[0098] In at least one embodiment, the data center may use CPUs, ASICs (application-specific integrated circuits), GPUs, FPGAs, or other hardware to perform training and / or inference using the aforementioned resources. Furthermore, one or more of the aforementioned software and / or hardware resources may be configured as a service that allows users to train or perform inference of information, such as image recognition, speech recognition, or other artificial intelligence services.

[0099] Logic (715) is used to perform inference and / or training operations associated with one or more embodiments. Details regarding the logic (715) are provided herein together with FIG. 7a and / or FIG. 7b. In at least one embodiment, the logic (715) may be used in a data center (900) to infer or predict operations based, at least partially, on weight parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0100] In at least one embodiment, at least one component illustrated or described in connection with FIGS. 7 through 9 is used to implement the techniques and / or functions described in connection with FIGS. 1 through 6. In at least one embodiment, logic (715) is used to perform image classification using input data comprising a voxel representation of an environment, based at least partially on a data structure for indicating the voxels to be generated during the conversion to the corresponding voxel representation of the point cloud, as described together with FIG. 1 and as otherwise described in this specification.

[0101] autonomous vehicles

[0102] FIG. 10a illustrates an example of an autonomous vehicle (1000) according to at least one embodiment. In at least one embodiment, the autonomous vehicle (1000) (alternatively referred to herein as “vehicle (1000)”) may be a passenger vehicle, such as a passenger car, truck, bus, and / or other type of vehicle that accommodates one or more passengers, without limitation. In at least one embodiment, the vehicle (1000) may be a semi-tractor-trailer truck used to transport cargo. In at least one embodiment, the vehicle (1000) may be an airplane, a robotic vehicle, or another type of vehicle.

[0103] Autonomous vehicles may be described in terms of levels of automation defined by the National Highway Traffic Safety Administration (“NHTSA”), a division of the US Department of Transportation, and the Society of Automotive Engineers (“SAE”) “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles” (e.g., Standard No. J3016-201806 published June 15, 2018; Standard No. J3016-201609 published September 30, 2016; and previous and future versions of these standards). In at least one embodiment, the vehicle (1000) may be capable of functionality according to one or more of Levels 1 through 5 of autonomous driving levels. For example, in at least one embodiment, the vehicle (1000) may be capable of conditional automation (Level 3), high level of automation (Level 4), and / or full automation (Level 5), depending on the embodiment.

[0104] In at least one embodiment, the vehicle (1000) may include, without limitation, components such as a chassis, a body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components of the vehicle. In at least one embodiment, the vehicle (1000) may include, without limitation, a propulsion system (1050), such as an internal combustion engine, a hybrid electric power plant, an all-electric engine, and / or other types of propulsion systems. In at least one embodiment, the propulsion system (1050) may be connected to a drive train of the vehicle (1000), which may, without limitation, include a transmission that enables propulsion of the vehicle (1000). In at least one embodiment, the propulsion system (1050) may be controlled in response to receiving signals from a throttle / accelerator(s) (1052).

[0105] In at least one embodiment, a steering system (1054), which may include, without limitation, a steering wheel, is used to steer the vehicle (1000) (e.g., along a desired path or route) when the propulsion system (1050) is operating (e.g., when the vehicle (1000) is moving). In at least one embodiment, the steering system (1054) may receive signals from steering actuator(s) (1056). In at least one embodiment, the steering wheel may be optional for full automation (Level 5) functionality. In at least one embodiment, a brake sensor system (1046) may be used to actuate the vehicle brakes in response to receiving signals from brake actuator(s) (1048) and / or brake sensors.

[0106] In at least one embodiment, a controller(s) (1036), which may include, without limitation, one or more "SoCs" (system on chips) (not shown in FIG. 10a) and / or "GPU(s)" (graphics processing unit(s)), provides signals (e.g., representing commands) to one or more components and / or systems of a vehicle (1000). For example, in at least one embodiment, the controller(s) (1036) may transmit signals to actuate vehicle brakes via brake actuator(s) (1048), actuate a steering system (1054) via steering actuator(s) (1056), and actuate a propulsion system (1050) via throttle / accelerator(s) (1052). In at least one embodiment, the controller(s) (1036) may include one or more onboard (e.g., integrated) computing devices for processing sensor signals, and output action commands (e.g., signals representing commands) that enable autonomous driving and / or assist a human driver in driving the vehicle (1000). In at least one embodiment, the controller(s) (1036) may include a first controller for autonomous driving functions, a second controller for functional safety functions, a third controller for artificial intelligence functions (e.g., computer vision), a fourth controller for infotainment functions, a fifth controller for redundancy in emergency situations, and / or other controllers. In at least one embodiment, a single controller may handle two or more of the above functions, two or more controllers may handle a single function, and / or any combination thereof.

[0107] In at least one embodiment, the controller(s) (1036) provide signals for controlling one or more components and / or systems of the vehicle (1000) in response to sensor data (e.g., sensor inputs) received from one or more sensors. In at least one embodiment, sensor data is, for example and without limitation, "GNSS" (global navigation satellite systems) sensor(s) (1058) (e.g., Global Positioning System sensor(s)), RADAR sensor(s) (1060), ultrasonic sensor(s) (1062), LIDAR sensor(s) (1064), "IMU" (inertial measurement unit) sensor(s) (1066) (e.g., accelerometer(s), gyroscope(s), magnetic compass or magnetic compasses, magnetometer(s), etc.), microphone(s) (1096), stereo camera(s) (1068), wide-view camera(s) (1070) (e.g., fisheye camera), infrared camera(s) (1072), surround camera(s) (1074) (e.g., 360-degree camera), long-range camera (not shown in FIG. 10a), medium-range camera(s) (not shown in FIG. 10a), It may be received from speed sensor(s) (1044) (for measuring the speed of a vehicle (1000), vibration sensor(s) (1042), steering sensor(s) (1040), brake sensor(s) (for example, as part of a brake sensor system (1046)), and / or other sensor types.

[0108] In at least one embodiment, one or more of the controller(s) (1036) receive inputs (e.g., represented by input data) from the device cluster (1032) of the vehicle (1000) and may provide outputs (e.g., represented by output data, display data, etc.) through an HMI (human-machine interface) display (1034), an auditory alerter, a loudspeaker, and / or other components of the vehicle (1000). In at least one embodiment, the outputs may include information such as vehicle speed, velocity, time, map data (e.g., a high-resolution map (not shown in FIG. 10a)), location data (e.g., the location of the vehicle (1000) as on a map), direction, the location of other vehicles (e.g., an occupancy grid), information regarding objects and the state of objects as perceived by the controller(s) (1036). For example, in at least one embodiment, the HMI display (1034) may display information regarding the presence of one or more objects (e.g., street signs, warning signs, traffic light changes, etc.) and / or information regarding driving operations that have been, are being, or will be done (e.g., changing lanes now, taking Exit 34B within 2 miles, etc.).

[0109] In at least one embodiment, the vehicle (1000) further includes a network interface (1024) capable of using wireless antenna(s) (1026) and / or modem(s) to communicate through one or more networks. For example, in at least one embodiment, the network interface (1024) can communicate through "LTE" (Long-Term Evolution), "WCDMA" (Wideband Code Division Multiple Access), "UMTS" (Universal Mobile Telecommunications System), "GSM" (Global System for Mobile communication), IMT-CDMA multi-carrier ("CDMA2000") networks, etc. In at least one embodiment, the wireless antenna(s) (1026) can also enable communication between objects in an environment (e.g., vehicles, mobile devices, etc.) using local area network(s) such as Bluetooth, Bluetooth "LE" (Low Energy), Z-Wave, ZigBee, etc., and / or "LPWANs" (low power wide-area network(s)) protocols such as LoRaWAN, SigFox, etc.

[0110] Logic (715) is used to perform inference and / or training operations associated with one or more embodiments. Details regarding the logic (715) are provided herein together with FIG. 7a and / or FIG. 7b. In at least one embodiment, the logic (715) may be used in a vehicle (1000) to infer or predict operations based at least partially on weight parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0111] FIG. 10b illustrates examples of camera positions and views for the autonomous vehicle (1000) of FIG. 10a according to at least one embodiment. In at least one embodiment, the cameras and their respective views are exemplary embodiments and are not intended to be limiting. For example, in at least one embodiment, additional and / or alternative cameras may be included and / or the cameras may be located at different positions on the vehicle (1000).

[0112] In at least one embodiment, camera types for the cameras may include digital cameras that are adaptable for use with components and / or systems of the vehicle (1000), but are not limited thereto. In at least one embodiment, the camera(s) may operate at "ASIL (automotive safety integrity level) B and / or other ASILs. In at least one embodiment, the camera types may be capable of any image capture rate, such as 60 fps (frames per second), 1220 fps, 240 fps, etc., depending on the embodiment. In at least one embodiment, the cameras may use a rolling shutter, a global shutter, other types of shutters, or a combination thereof. In at least one embodiment, the color filter array may include an "RCCC (red clear clear clear) color filter array, an "RCCB (red clear clear blue) color filter array," an "RBGC (red blue green clear) color filter array," a Foveon X3 color filter array, a Bayer sensor ("RGGB") color filter array, a monochromatic sensor color filter array, and / or other types of color filter arrays. In at least one embodiment, transparent pixel cameras, such as cameras with RCCC, RCCB, and / or RBGC color filter arrays, may be used in an effort to increase light sensitivity.

[0113] In at least one embodiment, one or more of the camera(s) may be used to perform "ADAS" (advanced driver assistance systems) functions (e.g., as part of a redundant or fail-safe design). For example, in at least one embodiment, a multi-functional mono-camera may be installed to provide functions including lane departure warning, traffic sign assistance, and intelligent headlamp control. In at least one embodiment, one or more of the camera(s) (e.g., all cameras) may simultaneously record and provide image data (e.g., video).

[0114] In at least one embodiment, one or more cameras may be mounted in a mounting assembly, such as a custom-designed (three-dimensional ("3D") printed) assembly, to block stray light and reflections from within the vehicle (1000) that may interfere with camera image data capture capabilities (e.g., reflections from the dashboard reflected in the windshield mirrors). Referring to wing-mirror mounting assemblies, in at least one embodiment, the wing-mirror assemblies may be custom 3D printed so that the camera mounting plate matches the shape of the wing-mirror. In at least one embodiment, the camera(s) may be integrated into the wing mirrors. In at least one embodiment, for side-view cameras, the camera(s) may also be integrated within the four pillars at each corner of the cabin.

[0115] In at least one embodiment, cameras having a field of view including parts of the environment in front of the vehicle (1000) (e.g., front cameras) may be used for a surround view to help identify forward-direction paths and obstacles, as well as to help provide important information for generating an occupancy grid and / or determining preferred vehicle paths with the help of one or more of the controller(s) (1036) and / or control SoCs. In at least one embodiment, front cameras may be used to perform many ADAS functions similar to LIDAR, including, without limitation, emergency braking, pedestrian detection, and collision avoidance. In at least one embodiment, front cameras may also be used for ADAS functions and systems, including, without limitation, other functions such as Lane Departure Warnings (LDW), Autonomous Cruise Control (ACC), and / or traffic sign recognition.

[0116] In at least one embodiment, various cameras may be used in the forward configuration, for example, including a monocular camera platform comprising a CMOS ("complementary metal oxide semiconductor") color imager. In at least one embodiment, a wide-view camera (1070) may be used to recognize objects entering the view from the surroundings (e.g., pedestrians, crossing traffic, or bicycles). Although only one wide-view camera (1070) is illustrated in FIG. 10b, in other embodiments, any number (including zero) of wide-view cameras may be present on the vehicle (1000). In at least one embodiment, any number of long-view camera(s) (1098) (e.g., a pair of long-view stereo cameras) may be used for depth-based object detection, particularly for objects for which the neural network has not yet been trained. In at least one embodiment, the long-view camera(s) (1098) may also be used for object detection and classification, as well as for basic object tracking.

[0117] In at least one embodiment, any number of stereo camera(s) (1068) may also be included in the forward configuration. In at least one embodiment, one or more of the stereo camera(s) (1068) may include an integrated control unit comprising a scalable processing unit capable of providing a multi-core microprocessor and programmable logic ("FPGA") with a "CAN (Controller Area Network)" or Ethernet interface integrated on a single chip. In at least one embodiment, such a unit may be used to generate a 3D map of the environment of the vehicle (1000) including distance estimates for all points in the image. In at least one embodiment, one or more of the stereo camera(s) (1068) may, without limitation, include a compact stereo vision sensor(s) that may include two camera lenses (one on the left and one on the right), and an image processing chip capable of measuring the distance from the vehicle (1000) to a target object and using generated information (e.g., metadata) to enable autonomous emergency braking and lane departure warning functions. In at least one embodiment, in addition to those described herein, or alternatively, other types of stereo camera(s) (1068) may be used.

[0118] In at least one embodiment, cameras having a field of view that includes parts of the environment for the sides of the vehicle (1000) (e.g., side-view cameras) may be used for a surround view and thus provide information used to generate side impact collision warnings as well as to generate and update an occupancy grid. For example, in at least one embodiment, surround camera(s) (1074) (e.g., four surround cameras as illustrated in FIG. 10b) may be positioned on the vehicle (1000). In at least one embodiment, the surround camera(s) (1074) may include, without limitation, any number and combination of wide-view cameras, fisheye camera(s), 360-degree camera(s), and / or similar cameras. For example, in at least one embodiment, four fisheye cameras may be positioned on the front, rear, and sides of the vehicle (1000). In at least one embodiment, the vehicle (1000) may use three surround camera(s) (1074) (e.g., left, right, and rear) and may utilize one or more other camera(s) (e.g., front camera) as a fourth surround-view camera.

[0119] In at least one embodiment, cameras having a field of view including parts of the environment behind the vehicle (1000) (e.g., rear-view cameras) may be used for parking assistance, surround view, rear collision warning, and the generation and updating of an occupancy grid. In at least one embodiment, as described herein, but not limited thereto, a wide variety of cameras may be used, including cameras also suitable as front camera(s) (e.g., long-range cameras (1098) and / or medium-range camera(s) (1076), stereo camera(s) (1068), infrared camera(s) (1072), etc.).

[0120] FIG. 10c is a block diagram illustrating an exemplary system architecture for the autonomous vehicle (1000) of FIG. 10a according to at least one embodiment. In at least one embodiment, each of the components, features, and systems of the vehicle (1000) of FIG. 10c is illustrated as being connected via a bus (1002). In at least one embodiment, the bus (1002) may, without limitation, include a CAN data interface (alternatively referred to herein as a “CAN bus”). In at least one embodiment, the CAN may be a network within the vehicle (1000) used to assist in controlling various features and functions of the vehicle (1000), such as the operation of brakes, acceleration, braking, steering, windshield wipers, etc. In at least one embodiment, the bus (1002) may be configured to have tens or even hundreds of nodes, each having its own unique identifier (e.g., CAN ID). In at least one embodiment, the bus (1002) can be read to detect steering wheel angle, ground speed, engine RPMs (revolutions per minute), button positions, and / or other vehicle status indicators. In at least one embodiment, the bus (1002) may be an ASIL B compliant CAN bus.

[0121] In at least one embodiment, in addition to CAN or alternatively, FlexRay and / or Ethernet protocols may be used. In at least one embodiment, any number of buses forming the bus (1002) may exist, which may include, without limitation, zero or more CAN buses, zero or more FlexRay buses, zero or more Ethernet buses, and / or zero or more other types of buses using different protocols. In at least one embodiment, two or more buses may be used to perform different functions and / or may be used for redundancy. For example, the first bus may be used for collision avoidance functionality, and the second bus may be used for operation control. In at least one embodiment, each bus of the bus (1002) may communicate with any of the components of the vehicle (1000), and two or more buses of the bus (1002) may communicate with corresponding components. In at least one embodiment, any number of “SoC(s)” (system(s) on chip(s)) (1004) (such as SoC(1004(A)) and SoC(1004(B)), each controller(s) (1036), and / or each computer in the vehicle may have access to the same input data (e.g., inputs from sensors of the vehicle (1000)) and may be connected to a common bus, such as a CAN bus.

[0122] In at least one embodiment, the vehicle (1000) may include one or more controller(s) (1036), such as those described herein in connection with FIG. 10a. In at least one embodiment, the controller(s) (1036) may be used for various functions. In at least one embodiment, the controller(s) (1036) may be coupled to any of the various other components and systems of the vehicle (1000) and may be used for the control of the vehicle (1000), artificial intelligence of the vehicle (1000), infotainment for the vehicle (1000), and / or other functions.

[0123] In at least one embodiment, the vehicle (1000) may include any number of SoCs (1004). In at least one embodiment, each of the SoCs (1004) may include, without limitation, "CPU(s)" (central processing units) (1006), "GPU(s)" (graphics processing units) (1008), processor(s) (1010), cache(s) (1012), accelerator(s) (1014), data storage(s) (1016), and / or other components and features not exemplified. In at least one embodiment, the SoC(s) (1004) may be used to control the vehicle (1000) on various platforms and systems. For example, in at least one embodiment, the SoC(s) (1004) may be combined with a “HD” (High Definition) map (1022) capable of obtaining map refreshes and / or updates from one or more servers (not shown in FIG. 10c) via a network interface (1024) and a system (e.g., a system of a vehicle (1000)).

[0124] In at least one embodiment, the CPU(s) (1006) may include a CPU cluster or CPU complex (alternatively referred to herein as "CCPLEX"). In at least one embodiment, the CPU(s) (1006) may include multiple cores and / or "L2" (level two) caches. For example, in at least one embodiment, the CPU(s) (1006) may include eight cores in a coherent multi-processor configuration. In at least one embodiment, the CPU(s) (1006) may include four dual-core clusters, each having a dedicated L2 cache (e.g., 2 MB (megabyte) L2 cache). In at least one embodiment, the CPU(s) (1006) (e.g., CCPLEX) may be configured to support concurrent cluster operations that enable any combination of the clusters of the CPU(s) (1006) to be active at any given time.

[0125] In at least one embodiment, one or more of the CPU(s) (1006) may implement power management capabilities including, without limitation, one or more of the following features: individual hardware blocks may be automatically clock-gated when idle to save dynamic power; each core clock may be gated when these cores are not actively executing instructions due to the execution of "WFI (Wait for Interrupt) / "WFE (Wait for Event) instructions; each core may be power-gated independently; each core cluster may be clock-gated independently when all cores are clock-gated or power-gated; and / or each core cluster may be power-gated independently when all cores are power-gated. In at least one embodiment, the CPU(s) (1006) may further implement an enhanced algorithm for managing power states, wherein allowed power states and expected wake-up times are specified, and hardware / microcode determines which best power state to enter for the core, cluster, and CCPLEX. In at least one embodiment, the processing cores may support simplified power state entry sequences in software as tasks offloaded to the microcode.

[0126] In at least one embodiment, the GPU(s) (1008) may include an integrated GPU (alternatively referred to herein as "iGPU"). In at least one embodiment, the GPU(s) (1008) may be programmable and efficient for parallel workloads. In at least one embodiment, the GPU(s) (1008) may use an enhanced tensor instruction set. In at least one embodiment, the GPU(s) (1008) may include one or more streaming microprocessors, each of which may include an L1 (level one) cache (e.g., an L1 cache having at least 96 KB of storage capacity), and two or more streaming microprocessors may share an L2 cache (e.g., an L2 cache having 512 KB of storage capacity). In at least one embodiment, the GPU(s) (1008) may include at least eight streaming microprocessors. In at least one embodiment, the GPU(s) (1008) may use computing API(s) (application programming interface(s)). In at least one embodiment, the GPU(s) (1008) may use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA model).

[0127] In at least one embodiment, one or more of the GPU(s) (1008) may be power-optimized for best performance in automotive and embedded use cases. For example, in at least one embodiment, the GPU(s) (1008) may be manufactured on FinFET (Fin field-effect transistor) circuits. In at least one embodiment, each streaming microprocessor may incorporate a number of mixed-precision processing cores partitioned into a number of blocks. For example, and without limitation, 64 PF32 cores and 32 FP64 cores may be partitioned into 4 processing blocks. In at least one embodiment, each processing block may be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, 2 mixed-precision NVIDIA Tensor Cores for deep learning matrix arithmetic, a level zero ("L0") instruction cache, a scheduler (e.g., a warp scheduler) or sequencer, a dispatch unit, and / or a 64 KB register file. In at least one embodiment, streaming microprocessors may include independent parallel integer and floating-point data paths to provide efficient execution of workloads with a mix of computing and addressing calculations. In at least one embodiment, streaming microprocessors may include independent thread scheduling capabilities to enable finer-grained synchronization and cooperation between parallel threads. In at least one embodiment, streaming microprocessors may include a combinational L1 data cache and a shared memory unit to improve performance while simplifying programming.

[0128] In at least one embodiment, one or more of the GPU(s) (1008) may include a “HBM” (high bandwidth memory) and / or 16 GB HBM2 memory subsystem to provide a peak memory bandwidth of about 900 GB / sec in some examples. In at least one embodiment, in addition to HBM memory or alternatively, “SGRAM” (synchronous graphics random-access memory), such as “GDDR5” (graphics double data rate type five synchronous random-access memory), may be used.

[0129] In at least one embodiment, the GPU(s) (1008) may include integrated memory technology. In at least one embodiment, "ATS" (address translation services) support may be used to allow the GPU(s) (1008) to directly access the page tables of the CPU(s) (1006). In at least one embodiment, when the GPU of the embodiment, GPU(s) (1008) "MMU" (memory management unit) experiences a miss, an address translation request may be sent to the CPU(s) (1006). In response, in at least one embodiment, two of the CPU(s) (1006) may look up their page tables for virtual-to-physical mapping of the address and send the translation back to the GPU(s) (1008). In at least one embodiment, the integrated memory technology may allow a single integrated virtual address space for the memory of both the CPU(s) (1006) and the GPU(s) (1008), thereby simplifying GPU(s) (1008) programming and the porting of applications to the GPU(s) (1008).

[0130] In at least one embodiment, the GPU(s) (1008) may include any number of access counters capable of keeping track of the frequency of access by the GPU(s) (1008) to the memory of other processors. In at least one embodiment, the access counter(s) may help ensure that memory pages are moved to the physical memory of the processor that accesses the pages most frequently, thereby improving efficiency for memory ranges shared between processors.

[0131] In at least one embodiment, one or more of the SoC(s) (1004) may include any number of cache(s) (1012) including those described herein. For example, in at least one embodiment, the cache(s) (1012) may include an "L3" (level three) cache available to both the CPU(s) (1006) and the GPU(s) (1008) (e.g., connected to the CPU(s) (1006) and the GPU(s) (1008). In at least one embodiment, the cache(s) (1012) may include a write-back cache capable of keeping track of the states of lines, such as by using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). In at least one embodiment, the L3 cache may include 4 MB or more of memory, depending on the embodiment, but smaller cache sizes may be used.

[0132] In at least one embodiment, one or more of the SoC(s) (1004) may include one or more accelerator(s) (1014) (e.g., hardware accelerators, software accelerators, or a combination thereof). In at least one embodiment, the SoC(s) (1004) may include a hardware acceleration cluster that may include optimized hardware accelerators and / or large on-chip memory. In at least one embodiment, large on-chip memory (e.g., 4 MB of SRAM) may enable the hardware acceleration cluster to accelerate neural networks and other computations. In at least one embodiment, the hardware acceleration cluster may be used to complement the GPU(s) (1008) and to offload some of the tasks of the GPU(s) (1008) (e.g., to free up more cycles of the GPU(s) (1008) to perform other tasks). In at least one embodiment, the accelerator(s) (1014) may be used for targeted workloads that are stable enough to adapt to acceleration (e.g., cognition, "CNNs" (convolutional neural networks), "RNNs" (recurrent neural networks), etc.). In at least one embodiment, the CNN may include region-based or "RCNNs" (regional convolutional neural networks) and fast RCNNs (e.g., as used for object detection) or other types of CNNs.

[0133] In at least one embodiment, the accelerator(s) (1014) (e.g., a hardware acceleration cluster) may include one or more "DLAs" (deep learning accelerators). In at least one embodiment, the DLA(s) may include, without limitation, one or more "TPUs" (Tensor processing units) that may be configured to provide an additional 10 trillion operations per second for deep learning applications and inference. In at least one embodiment, the TPUs may be accelerators configured to perform image processing functions (e.g., for CNNs, RCNNs, etc.) and optimized for this purpose. In at least one embodiment, the DLA(s) may be further optimized for inference as well as for a specific set of neural network types and floating-point operations. In at least one embodiment, the design of the DLA(s) may provide more performance per millimeter than a conventional general-purpose GPU and typically significantly exceeds the performance of a CPU. In at least one embodiment, the TPU(s) may perform some functions, including single-instance convolution functions that support post-processor functions as well as INT8, INT16, and FP16 data types for both features and weights, for example. In at least one embodiment, the DLA(s) may rapidly and efficiently execute neural networks, particularly CNNs, on processed or unprocessed data for any of various functions, which include, for example and without limitation, the following: a CNN for object identification and detection using data from camera sensors; a CNN for distance estimation using data from camera sensors; a CNN for emergency vehicle detection and identification using data from microphones; a CNN for face recognition and vehicle owner identification using data from camera sensors; and / or a CNN for security and / or safety-related events.

[0134] In at least one embodiment, the DLA(s) can perform any function of the GPU(s) (1008), and by using an inference accelerator, for example, the designer can target the DLA(s) or the GPU(s) (1008) for any function. For example, in at least one embodiment, the designer can concentrate the processing of CNNs and floating-point operations on the DLA(s) and leave other functions to the GPU(s) (1008) and / or accelerator(s) (1014).

[0135] In at least one embodiment, the accelerator(s) (1014) may include a "programmable vision accelerator" (PVA), which may alternatively be referred to herein as a computer vision accelerator. In at least one embodiment, the PVA may be designed and configured to accelerate computer vision algorithms for "advanced driver assistance systems" (ADAS) (1038), autonomous driving, "augmented reality" (AR) applications, and / or "virtual reality" (VR) applications. In at least one embodiment, the PVA may provide a balance between performance and flexibility. For example, in at least one embodiment, each PVA may include, for example and without limitation, any number of "RISC" (reduced instruction set computer) cores, "DMA" (direct memory access), and / or any number of vector processors.

[0136] In at least one embodiment, RISC cores may interact with image sensors (e.g., image sensors of any cameras described herein), image signal processor(s), etc. In at least one embodiment, each RISC core may include any amount of memory. In at least one embodiment, RISC cores may use any of a number of protocols, depending on the embodiment. In at least one embodiment, RISC cores may run a "RTOS" (real-time operating system). In at least one embodiment, RISC cores may be implemented using one or more integrated circuit devices, "ASICs" (application-specific integrated circuits), and / or memory devices. For example, in at least one embodiment, RISC cores may include an instruction cache and / or tightly coupled RAM.

[0137] In at least one embodiment, the DMA may enable the components of the PVA to access system memory independently of the CPU(s) (1006). In at least one embodiment, the DMA may support any number of features used to provide optimization for the PVA, including, but not limited to, support for multidimensional addressing and / or circular addressing. In at least one embodiment, the DMA may support at least six dimensions of addressing, including, without limitation, block width, block height, block depth, horizontal block stepping, vertical block stepping, and / or depth stepping.

[0138] In at least one embodiment, vector processors may be programmable processors designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing capabilities. In at least one embodiment, the PVA may include a PVA core and two vector processing subsystem partitions. In at least one embodiment, the PVA core may include a processor subsystem, DMA engine(s) (e.g., two DMA engines), and / or other peripherals. In at least one embodiment, the vector processing subsystem may operate as the main processing engine of the PVA and may include a "VPU (vector processing unit)," an instruction cache, and / or vector memory (e.g., "VMEM"). In at least one embodiment, the VPU core may include a digital signal processor, such as a SIMD (single instruction, multiple data) or VLIW (very long instruction word) digital signal processor. In at least one embodiment, a combination of SIMD and VLIW may enhance throughput and speed.

[0139] In at least one embodiment, each of the vector processors may include an instruction cache and may be coupled to dedicated memory. As a result, in at least one embodiment, each of the vector processors may be configured to run independently of other vector processors. In at least one embodiment, the vector processors included in a particular PVA may be configured to utilize data parallelism. For example, in at least one embodiment, multiple vector processors included in a single PVA may execute a common computer vision algorithm on different regions of an image. In at least one embodiment, the vector processors included in a particular PVA may simultaneously execute different computer vision algorithms on a single image, or even execute different algorithms on sequential images or parts of an image. In at least one embodiment, among other things, any number of PVAs may be included in a hardware acceleration cluster, and any number of vector processors may be included in each PVA. In at least one embodiment, the PVA may include additional "ECC (error correcting code) memory" to enhance overall system safety.

[0140] In at least one embodiment, the accelerator(s) (1014) may include a computer vision network on-chip and static random-access memory (SRAM) to provide high-bandwidth, low-latency SRAM for the accelerator(s) (1014). In at least one embodiment, the on-chip memory may include at least 4 MB SRAM, for example and without limitation, containing eight field-configurable memory blocks that can be accessed by both the PVA and the DLA. In at least one embodiment, each pair of memory blocks may include an advanced peripheral bus (APB) interface, a configuration circuit, a controller, and a multiplexer. In at least one embodiment, any type of memory may be used. In at least one embodiment, the PVA and the DLA may access the memory through a backbone that provides high-speed access to the memory to the PVA and the DLA. In at least one embodiment, the backbone may include a computer vision network on-chip that interconnects the PVA and the DLA to the memory (for example, using APB).

[0141] In at least one embodiment, the computer vision network on-chip may include an interface that determines whether both the PVA and DLA provide ready and valid signals prior to the transmission of any control signal / address / data. In at least one embodiment, the interface may provide separate phases and separate channels for transmitting control signals / addresses / data, as well as burst-type communications for continuous data transmission. In at least one embodiment, the interface may comply with ISO (International Organization for Standardization) 26262 or IEC (International Electrotechnical Commission) 61508 standards, but other standards and protocols may be used.

[0142] In at least one embodiment, one or more of the SoC(s) (1004) may include a real-time ray-tracing hardware accelerator. In at least one embodiment, the real-time ray-tracing hardware accelerator may be used to rapidly and efficiently determine the locations and ranges of objects (e.g., within a world model), to generate real-time visualization simulations, to interpret RADAR signals, to synthesize and / or analyze sound propagation, to simulate SONAR systems, to simulate general wave propagation, to compare with LIDAR data for the purposes of localization and / or other functions, and / or for other uses.

[0143] In at least one embodiment, the accelerator(s) (1014) may have a wide range of applications for autonomous driving. In at least one embodiment, the PVA may be used for core processing stages in ADAS and autonomous vehicles. In at least one embodiment, the capabilities of the PVA are a good match for algorithmic domains requiring predictable processing at low power and low latency. That is, the PVA performs well for semi-dense or dense normal computing, even for small data sets, which may require predictable runtimes at low latency and low power. In at least one embodiment, as in the vehicle (1000), the PVAs may be designed to execute classical computer vision algorithms, as they may be efficient in object detection and operate in integer mathematics.

[0144] For example, according to at least one embodiment of the technology, the PVA is used to perform computer stereo vision. In at least one embodiment, a semi-global matching-based algorithm may be used in some examples, but this is not intended to be limiting. In at least one embodiment, applications for Level 3-5 autonomous driving use motion estimation / stereo matching depending on the situation (e.g., structure from motion, pedestrian recognition, lane detection, etc.). In at least one embodiment, the PVA can perform computer stereo vision functions on inputs from two monocular cameras.

[0145] In at least one embodiment, the PVA may be used to perform dense optical flow. For example, in at least one embodiment, the PVA may process raw RADAR data (e.g., using a 4D Fast Fourier Transform) to provide processed RADAR data. In at least one embodiment, the PVA is used for time-of-flight depth processing by processing raw time-of-flight data to provide processed time-of-flight data.

[0146] In at least one embodiment, the DLA may be used to execute any type of network to enhance control and driving safety, including, for example and without limitation, a neural network that outputs a measure of confidence for each object detection. In at least one embodiment, confidence may be expressed or interpreted as a probability or as providing a relative "weight" for each detection compared to other detections. In at least one embodiment, the confidence measure enables the system to make further decisions regarding which detections should be considered as true positive detections rather than false positive detections. In at least one embodiment, the system may set a threshold for confidence and consider only detections that exceed the threshold as true positive detections. In an embodiment where an "AEB (automatic emergency braking) system" is used, false positive detections would cause the vehicle to automatically perform emergency braking, which is obviously undesirable. In at least one embodiment, highly reliable detections may be considered as triggers for AEB. In at least one embodiment, the DLA may execute a neural network to regress the confidence value. In at least one embodiment, the neural network may take, as its input, at least a subset of parameters such as bounding box dimensions, ground plane estimates obtained (e.g., from another subsystem), vehicle (1000) orientation, distance, and output from an IMU sensor(s) (1066) corresponding to 3D position estimates of an object obtained from the neural network and / or other sensors (e.g., LIDAR sensor(s) (1064) or RADAR sensor(s) (1060)).

[0147] In at least one embodiment, one or more of the SoC(s) (1004) may include data storage(s) (1016) (e.g., memory). In at least one embodiment, the data storage(s) (1016) may be on-chip memory of the SoC(s) (1004) capable of storing neural networks to be executed on the GPU(s) (1008) and / or DLA. In at least one embodiment, the data storage(s) (1016) may have a capacity large enough to store multiple instances of neural networks for redundancy and safety. In at least one embodiment, the data storage(s) (1016) may include L2 or L3 cache(s).

[0148] In at least one embodiment, one or more of the SoC(s) (1004) may include any number of processor(s) (1010) (e.g., embedded processors). In at least one embodiment, the processor(s) (1010) may include a boot and power management processor, which may be a dedicated processor and subsystem for handling boot power and management functions and related security enforcement. In at least one embodiment, the boot and power management processor may be part of the boot sequence of the SoC(s) (1004) and may provide runtime power management services. In at least one embodiment, the boot power and management processor may provide clock and voltage programming, assistance in system low-power state transitions, management of SoC(s) (1004) thermals and temperature sensors, and / or management of SoC(s) (1004) power states. In at least one embodiment, each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to the temperature, and the SoC(s) (1004) may use the ring oscillators to detect the temperatures of the CPU(s) (1006), GPU(s) (1008), and / or accelerator(s) (1014). In at least one embodiment, when it is determined that the temperatures exceed a threshold, the boot and power management processor may then enter a temperature failure routine to put the SoC(s) (1004) into a lower power state and / or put the vehicle (1000) into a chauffeur-safe shutdown mode (e.g., put the vehicle (1000) into a safe shutdown).

[0149] In at least one embodiment, the processor(s) (1010) may further include a set of embedded processors capable of serving as an audio processing engine that may be an audio subsystem enabling full hardware support for multi-channel audio through a plurality of interfaces and a wide and flexible range of audio I / O interfaces. In at least one embodiment, the audio processing engine is a dedicated processor core having a digital signal processor with dedicated RAM.

[0150] In at least one embodiment, the processor(s) (1010) may further include an always-on processor engine capable of providing hardware features necessary to support low-power sensor management and to enable use cases. In at least one embodiment, the always-on processor engine may include, without limitation, a processor core, tightly coupled RAM, supporting peripherals (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0151] In at least one embodiment, the processor(s) (1010) may further include a safety cluster engine, without limitation, comprising a dedicated processor subsystem for handling safety management for automotive applications. In at least one embodiment, the safety cluster engine may, without limitation, include two or more processor cores, tightly coupled RAM, supporting peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In safety mode, in at least one embodiment, the two or more cores may operate in lockstep mode and function as a single core having comparison logic to detect any differences between their operations. In at least one embodiment, the processor(s) (1010) may further include a real-time camera engine, without limitation, comprising a dedicated processor subsystem for handling real-time camera management. In at least one embodiment, the processor(s) (1010) may further include a high-dynamic-range signal processor, without limitation, comprising an image signal processor which is a hardware engine that is part of the camera processing pipeline.

[0152] In at least one embodiment, the processor(s) (1010) may include a video image synthesizer which may be a processing block (e.g., implemented on a microprocessor) that implements video post-processing functions required by a video playback application to produce a final image for a player window. In at least one embodiment, the video image synthesizer may perform lens distortion correction for wide-view camera(s) (1070), surround camera(s) (1074), and / or cabin-in-cabin monitoring camera sensor(s). In at least one embodiment, the cabin-in-cabin monitoring camera sensor(s) are preferably monitored by a neural network running on another instance of the SoC (1004) configured to identify cabin events and respond accordingly. In at least one embodiment, the cabin-in-cabin system may perform lip reading, without limitation, to enable cellular services and make phone calls, dictate emails, change the destination of the vehicle, enable or change the infotainment system and settings of the vehicle, or provide voice-enabled web surfing. In at least one embodiment, certain functions are available to the driver when the vehicle is operating in autonomous mode and otherwise disabled.

[0153] In at least one embodiment, the video image synthesizer may include enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, in at least one embodiment where motion occurs in the video, the noise reduction appropriately weights spatial information to reduce the weights of information provided by adjacent frames. In at least one embodiment where the image or part of the image does not contain motion, the temporal noise reduction performed by the video image synthesizer may use information from the previous image to reduce noise in the current image.

[0154] In at least one embodiment, the video image synthesizer may also be configured to perform stereo rectification on input stereo lens frames. In at least one embodiment, the video image synthesizer may be additionally used for user interface synthesis when the operating system desktop is in use, and the GPU(s) (1008) are not required to continuously render new surfaces. In at least one embodiment, when the GPU(s) (1008) are powered on and 3D rendering is active, the video image synthesizer may be used to offload the GPU(s) (1008) to improve performance and responsiveness.

[0155] In at least one embodiment, one or more of the SoC(s) (1004) may further include a video input block that can be used for a MIPI (mobile industry processor interface) camera serial interface, a high-speed interface, and / or camera and associated pixel input functions for receiving video and input from cameras. In at least one embodiment, one or more of the SoC(s) (1004) may further include an input / output controller(s) that can be controlled by software and can be used to receive uncommitted I / O signals.

[0156] In at least one embodiment, one or more of the SoC(s) (1004) may further include a wide range of peripheral interfaces that enable communication with peripherals, audio encoders / decoders ("codecs"), power management, and / or other devices. In at least one embodiment, the SoC(s) (1004) may be used to process data from cameras (e.g., connected via Gigabit Multimedia Serial Link and Ethernet channels), sensors (e.g., LIDAR sensor(s) (1064), RADAR sensor(s) (1060), etc. that can be connected via Ethernet channels), data from a bus (1002) (e.g., speed of the vehicle (1000), steering wheel position, etc.), data from GNSS sensor(s) (1058) (e.g., connected via an Ethernet bus or CAN bus), etc. In at least one embodiment, one or more of the SoC(s) (1004) may further include dedicated high-performance mass storage controllers that may include their own DMA engines and may be used to free the CPU(s) (1006) from routine data management tasks.

[0157] In at least one embodiment, the SoC(s) (1004) may be an end-to-end platform with a flexible architecture spanning automation levels 3-5, thereby providing a comprehensive functional safety architecture that provides a platform for a flexible, reliable driving software stack, along with deep learning tools, which utilizes and efficiently employs computer vision and ADAS techniques for diversity and redundancy. In at least one embodiment, the SoC(s) (1004) may be faster, more reliable, and much more energy-efficient and space-efficient than conventional systems. For example, in at least one embodiment, the accelerator(s) (1014), when combined with CPU(s) (1006), GPU(s) (1008), and data storage(s) (1016), may provide a fast and efficient platform for Level 3-5 autonomous vehicles.

[0158] In at least one embodiment, computer vision algorithms may be executed on CPUs that can be configured to execute a wide variety of processing algorithms across a wide variety of visual data using a high-level programming language such as C. However, in at least one embodiment, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption, for example. In at least one embodiment, many CPUs cannot execute complex object detection algorithms in real time, which are used in in-vehicle ADAS applications and actual Level 3-5 autonomous vehicles.

[0159] The embodiments described herein allow multiple neural networks to be executed simultaneously and / or sequentially, and allow the results to be combined together to enable Level 3-5 autonomous driving functionality. For example, in at least one embodiment, a CNN running on a DLA or individual GPU (e.g., GPU(s) (1020)) may include text and word recognition, thereby enabling the reading and understanding of traffic signs, including signs that the neural network has not specifically trained on. In at least one embodiment, the DLA may further include a neural network capable of identifying, interpreting, and providing a semantic understanding of the sign, and transmitting that semantic understanding to path planning modules running on a CPU complex.

[0160] In at least one embodiment, for Level 3, 4, or 5 driving, multiple neural networks may be executed simultaneously. For example, in at least one embodiment, a warning sign mentioning "Caution: flashing lights indicate icy conditions" along with electric lights may be interpreted independently or collectively by some neural networks. In at least one embodiment, this warning sign itself may be identified as a traffic sign by a first deployed neural network (e.g., a trained neural network), and the text "flashing lights indicate icy conditions" may be interpreted by a second deployed neural network, which notifies the vehicle's route planning software (preferably executed on a CPU complex) that icy conditions exist when flashing lights are detected. In at least one embodiment, flashing lights may be identified by operating a third deployed neural network over multiple frames, which notifies the vehicle's route planning software of the presence (or absence) of flashing lights. In at least one embodiment, all three neural networks can be executed simultaneously, such as in the DLA and / or on the GPU(s) (1008).

[0161] In at least one embodiment, a CNN for face recognition and vehicle owner identification may use data from camera sensors to identify the presence of an authorized driver and / or owner of the vehicle (1000). In at least one embodiment, an always-on sensor processing engine may be used to unlock the vehicle when the owner approaches the driver's door and turns on the lights, and, in security mode, to disable the vehicle when the owner leaves the vehicle. In this way, the SoC(s) (1004) provide security against theft and / or vehicle hijacking.

[0162] In at least one embodiment, the CNN for emergency vehicle detection and identification may use data from microphones (1096) to detect and identify emergency vehicle sirens. In at least one embodiment, the SoC(s) (1004) use the CNN to classify visual data as well as environmental and city sounds. In at least one embodiment, the CNN running on the DLA is trained to identify the relative closing speed of emergency vehicles (e.g., by using Doppler effects). In at least one embodiment, the CNN may also be trained to identify specific emergency vehicles in the local area where the vehicle is operating, as identified by the GNSS sensor(s) (1058). In at least one embodiment, when operating in Europe, the CNN will attempt to detect European sirens, and when operating in North America, the CNN will attempt to identify only North American sirens. In at least one embodiment, once an emergency vehicle is detected, a control program may be used to execute an emergency vehicle safety routine, and thus, until the emergency vehicles pass, with the assistance of the ultrasonic sensor(s) (1062), slow down the vehicle, stop to the side of the road, park the vehicle, and / or idle the vehicle.

[0163] In at least one embodiment, the vehicle (1000) may include CPU(s) (1018) (e.g., individual CPU(s), or dCPU(s)) which can be coupled to the SoC(s) (1004) via a high-speed interconnect (e.g., PCIe). In at least one embodiment, the CPU(s) (1018) may include, for example, an X86 processor. The CPU(s) (1018) may be used to perform any of various functions, for example, mediating potentially inconsistent results between ADAS sensors and the SoC(s) (1004), and / or monitoring the status and health of the controller(s) (1036) and / or the infotainment "infotainment SoC" (infotainment system on a chip) (1030). In at least one embodiment, the SoC(s) (1004) includes one or more interconnects, and the interconnects may include peripheral component interconnection express (PCIe).

[0164] In at least one embodiment, the vehicle (1000) may include GPU(s) (1020) (e.g., discrete GPU(s) or dGPU(s)) that can be coupled to the SoC(s) (1004) via a high-speed interconnect (e.g., NVIDIA’s NVLINK channel). In at least one embodiment, the GPU(s) (1020) may provide additional artificial intelligence functionality, such as by running redundant and / or different neural networks, and may be used to train and / or update neural networks based at least partially on inputs (e.g., sensor data) from the sensors of the vehicle (1000).

[0165] In at least one embodiment, the vehicle (1000) may further include a network interface (1024) which may include, without limitation, wireless antenna(s) (1026) (one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). In at least one embodiment, the network interface (1024) may be used to enable wireless connectivity with other vehicles and / or computing devices (e.g., passenger client devices) to Internet cloud services (e.g., by server(s) and / or other network devices). In at least one embodiment, to communicate with other vehicles, a direct link may be established between the vehicle (1000) and other vehicles and / or an indirect link may be established (e.g., through networks and via the Internet). In at least one embodiment, direct links may be provided using vehicle-to-vehicle communication links. In at least one embodiment, a vehicle-to-vehicle communication link may provide vehicle (1000) information regarding vehicles in close proximity to the vehicle (1000) (e.g., vehicles in front, to the side, and / or behind the vehicle (1000)). In at least one embodiment, this aforementioned functionality may be part of the cooperative adaptive cruise control functionality of the vehicle (1000).

[0166] In at least one embodiment, the network interface (1024) may include an SoC that provides modulation and demodulation capabilities and enables controller(s) (1036) to communicate over a wireless network. In at least one embodiment, the network interface (1024) may include a radio frequency front-end for up-conversion from baseband to radio frequency and down-conversion from radio frequency to baseband. In at least one embodiment, frequency conversions may be performed in any technically feasible manner. For example, frequency conversions may be performed through well-known processes and / or using super-heterodyne processes. In at least one embodiment, the radio frequency front-end functionality may be provided by a separate chip. In at least one embodiment, the network interfaces may include wireless capabilities for communicating over LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0167] In at least one embodiment, the vehicle (1000) may further include data storage(s) (1028) which may include off-chip (e.g., off SoC(s) (1004)) storage, without limitation. In at least one embodiment, the data storage(s) (1028) may include, without limitation, one or more storage elements including RAM, SRAM, DRAM (dynamic random-access memory), VRAM (video random-access memory), flash memory, hard disks, and / or other components and / or devices capable of storing at least one bit of data.

[0168] In at least one embodiment, the vehicle (1000) may further include GNSS sensor(s) (1058) (e.g., GPS and / or auxiliary GPS sensors) that assist in mapping, perception, occupancy grid generation, and / or path planning functions. In at least one embodiment, any number of GNSS sensor(s) (1058) may be used, for example and without limitation, including a GPS using a USB connector with an Ethernet-serial (e.g., RS-232) bridge.

[0169] In at least one embodiment, the vehicle (1000) may further include RADAR sensor(s) (1060). In at least one embodiment, the RADAR sensor(s) (1060) may be used by the vehicle (1000) for long-range vehicle detection, even in darkness and / or severe weather conditions. In at least one embodiment, the RADAR functional safety levels may be ASIL B. In at least one embodiment, the RADAR sensor(s) (1060) may use a CAN bus and / or bus (1002) for control and to access object tracking data (e.g., to transmit data generated by the RADAR sensor(s) (1060)), and in some examples, access Ethernet channels to access raw data. In at least one embodiment, a wide variety of RADAR sensor types may be used. For example, and without limitation, the RADAR sensor(s) (1060) may be suitable for front, rear, and side RADAR applications. In at least one embodiment, one or more of the RADAR sensor(s) (1060) are pulse Doppler RADAR sensors.

[0170] In at least one embodiment, the RADAR sensor(s) (1060) may include different configurations, such as long range with a narrow field of view, short range with a wide field of view, short range side coverage, etc. In at least one embodiment, long range RADAR may be used for adaptive cruise control functionality. In at least one embodiment, long range RADAR systems may provide a wide field of view realized by two or more independent scans, such as within a range of 250 m (meter). In at least one embodiment, the RADAR sensor(s) (1060) may help distinguish between static and moving objects and may be used by an ADAS system (1038) for emergency braking assistance and forward collision warning. In at least one embodiment, the sensors (1060(s)) included in the long range RADAR system may, without limitation, include a number (e.g., six or more) fixed RADAR antennas and a monostatic multimodal RADAR with high-speed CAN and FlexRay interfaces. In at least one embodiment, by means of six antennas, the central four antennas can generate a focused beam pattern designed to record the surroundings of the vehicle (1000) at higher speeds with minimal interference from traffic in adjacent lanes. In at least one embodiment, the other two antennas can extend the field of view and thus enable rapid detection of vehicles entering or leaving the lane of the vehicle (1000).

[0171] In at least one embodiment, medium-range RADAR systems may include a range of up to 160 m (forward) or 80 m (rear), and a field of view of up to 42 degrees (forward) or 150 degrees (rear), for example. In at least one embodiment, short-range RADAR systems may include, without limitation, any number of RADAR sensor(s) (1060) designed to be installed at both ends of the rear bumper. When installed at both ends of the rear bumper, in at least one embodiment, the RADAR sensor system may generate two beams that continuously monitor blind spots in the rearward direction and at the side of the vehicle. In at least one embodiment, short-range RADAR systems may be used in an ADAS system (1038) for blind spot detection and / or lane change assistance.

[0172] In at least one embodiment, the vehicle (1000) may further include ultrasonic sensor(s) (1062). In at least one embodiment, the ultrasonic sensor(s) (1062), which may be located at the front, rear, and / or side positions of the vehicle (1000), may be used for parking assistance and / or to generate and update an occupancy grid. In at least one embodiment, a wide variety of ultrasonic sensor(s) (1062) may be used, and different ultrasonic sensor(s) (1062) may be used for different detection ranges (e.g., 2.5 m, 4 m). In at least one embodiment, the ultrasonic sensor(s) (1062) may operate at ASIL B functional safety levels.

[0173] In at least one embodiment, the vehicle (1000) may include LIDAR sensor(s) (1064). In at least one embodiment, the LIDAR sensor(s) (1064) may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. In at least one embodiment, the LIDAR sensor(s) (1064) may operate at a functional safety level ASIL B. In at least one embodiment, the vehicle (1000) may include a plurality of LIDAR sensors (1064) (e.g., 2, 4, 6, etc.) capable of using an Ethernet channel (e.g., to provide data to a Gigabit Ethernet switch).

[0174] In at least one embodiment, the LIDAR sensor(s) (1064) may provide a list of objects and their distances for a 360-degree field of view. In at least one embodiment, a commercially available LIDAR sensor(s) (1064) may have an advertised range of approximately 100 m, for example, with an accuracy of 2 cm to 3 cm and support for a 100 Mbps Ethernet connection. In at least one embodiment, one or more non-protruding LIDAR sensors may be used. In this embodiment, the LIDAR sensor(s) (1064) may include a small device that can be embedded in the front, rear, side, and / or corner locations of the vehicle (1000). In at least one embodiment, the LIDAR sensor(s) (1064) may provide a field of view of up to 120 degrees horizontal and 35 degrees vertical, with a range of 200 m, even for low-reflectivity objects. In at least one embodiment, the front-mounted LIDAR sensor(s) (1064) may be configured for a horizontal field of view between 45 and 135 degrees.

[0175] In at least one embodiment, LIDAR technologies, such as 3D flash LIDAR, may also be used. In at least one embodiment, 3D flash LIDAR uses a laser flash as a transmission source to illuminate the surroundings of the vehicle (1000) up to approximately 200 m. In at least one embodiment, the flash LIDAR unit includes, without limitation, a receiver that records the laser pulse passage time and reflected light on each pixel, which eventually corresponds to the range from the vehicle (1000) to objects. In at least one embodiment, the flash LIDAR may allow highly accurate and distortion-free images of the surroundings to be generated by all laser flashes. In at least one embodiment, four flash LIDAR sensors may be placed on each side of the vehicle (1000). In at least one embodiment, 3D flash LIDAR systems include, without limitation, a solid-state 3D gaze array LIDAR camera (e.g., a non-scanning LIDAR device) that has no moving parts other than a pan. In at least one embodiment, the flash LIDAR device can use a 5 nanosecond class I (eye-safe) laser pulse per frame and can capture the reflected laser light as a 3D range point cloud and co-registered intensity data.

[0176] In at least one embodiment, the vehicle (1000) may further include IMU sensor(s) (1066). In at least one embodiment, the IMU sensor(s) (1066) may be located at the center of the rear axle of the vehicle (1000). In at least one embodiment, the IMU sensor(s) (1066) may include accelerometer(s), magnetometer(s), gyroscope(s), magnetic compass, magnetic compass, and / or other sensor types, for example and without limitation. In at least one embodiment, such as in 6-axis applications, the IMU sensor(s) (1066) may include accelerometers and gyroscopes, without limitation. In at least one embodiment, such as in 9-axis applications, the IMU sensor(s) (1066) may include accelerometers, gyroscopes, and magnetometers, without limitation.

[0177] In at least one embodiment, the IMU sensor(s) (1066) may be implemented as a miniature, high-performance GPS / INS (GPS-Aided Inertial Navigation System) that combines MEMS (micro-electro-mechanical systems) inertial sensors, a high-sensitivity GPS receiver, and an advanced Kalman filtering algorithm to provide estimates of position, speed, and attitude. In at least one embodiment, the IMU sensor(s) (1066) may enable the vehicle (1000) to estimate its direction of travel without requiring input from a magnetic sensor by directly observing and correlating changes in speed from the GPS to the IMU sensor(s) (1066). In at least one embodiment, the IMU sensor(s) (1066) and the GNSS sensor(s) (1058) may be combined into a single integrated unit.

[0178] In at least one embodiment, the vehicle (1000) may include microphone(s) (1096) placed on and / or around the vehicle (1000). In at least one embodiment, the microphone(s) (1096) may be used, among other things, for emergency vehicle detection and identification.

[0179] In at least one embodiment, the vehicle (1000) may further include any number of camera types, including stereo camera(s) (1068), wide-view camera(s) (1070), infrared camera(s) (1072), surround camera(s) (1074), long-range camera(s) (1098), medium-range camera(s) (1076), and / or other camera types. In at least one embodiment, cameras may be used to capture image data around the entire periphery of the vehicle (1000). In at least one embodiment, the types of cameras used depend on the vehicle (1000). In at least one embodiment, any combination of camera types may be used to provide the necessary coverage around the vehicle (1000). In at least one embodiment, the number of cameras deployed may vary depending on the embodiment. For example, in at least one embodiment, the vehicle (1000) may include six cameras, seven cameras, ten cameras, twelve cameras, or any other number of cameras. In at least one embodiment, the cameras may support GMSL (Gigabit Multimedia Serial Link) and / or Gigabit Ethernet communication, for example and without limitation. In at least one embodiment, each camera may be as previously described in more detail in connection with FIGS. 10a and FIGS. 10b.

[0180] In at least one embodiment, the vehicle (1000) may further include vibration sensor(s) (1042). In at least one embodiment, the vibration sensor(s) (1042) may measure vibrations of components of the vehicle (1000), such as axle(s). For example, in at least one embodiment, changes in vibrations may indicate changes in road surfaces. In at least one embodiment, when two or more vibration sensors (1042) are used, differences between vibrations may be used to determine friction or slippage of the road surface (e.g., when there is a difference in vibration between a power-driven axle and a freely rotating axle).

[0181] In at least one embodiment, the vehicle (1000) may include an ADAS system (1038). In at least one embodiment, the ADAS system (1038) may include, in some examples, an SoC, without limitation. In at least one embodiment, the ADAS system (1038) may include, without limitation, any number and combination of autonomous / adaptive / automatic cruise control ("ACC") systems, cooperative adaptive cruise control ("CACC") systems, forward collision warning ("FCW") systems, automatic emergency braking ("AEB") systems, lane departure warning ("LDW") systems, lane keeping assist ("LKA") systems, blind spot warning ("BSW") systems, rear cross-traffic warning ("RCTW") systems, collision warning ("CW") systems, lane centering ("LC") systems, and / or other systems, features, and / or functionalities.

[0182] In at least one embodiment, the ACC system may use RADAR sensor(s) (1060), LIDAR sensor(s) (1064), and / or any number of camera(s). In at least one embodiment, the ACC system may include a longitudinal ACC system and / or a lateral ACC system. In at least one embodiment, the longitudinal ACC system monitors and controls the distance to another vehicle immediately ahead of the vehicle (1000) and automatically adjusts the speed of the vehicle (1000) to maintain a safe distance from the vehicles ahead. In at least one embodiment, the lateral ACC system performs distance maintenance and advises the vehicle (1000) to change lanes when necessary. In at least one embodiment, the lateral ACC is related to other ADAS applications, such as LC and CW.

[0183] In at least one embodiment, the CACC system uses information from other vehicles that can be received from other vehicles via a network connection (e.g., via the Internet), indirectly, or via a wireless link through a wireless antenna(s) (1026) and / or a network interface (1024). In at least one embodiment, direct links may be provided by a "V2V" (vehicle-to-vehicle) communication link, while indirect links may be provided by an "I2V" (infrastructure-to-vehicle) communication link. Generally, V2V communication provides information regarding vehicles immediately preceding (e.g., vehicles immediately ahead of vehicle (1000) and in the same lane as the vehicle), whereas I2V communication provides information regarding traffic further ahead. In at least one embodiment, the CACC system may include either or both of I2V and V2V information sources. In at least one embodiment, if information about the vehicles ahead of the vehicle (1000) is provided, the CACC system may be more reliable and has the potential to improve traffic flow and reduce congestion on the road.

[0184] In at least one embodiment, the FCW system is designed to alert the driver to danger so that the driver can take corrective action. In at least one embodiment, the FCW system uses a front camera and / or RADAR sensor(s) (1060) coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to provide driver feedback, such as a display, speaker, and / or vibration component. In at least one embodiment, the FCW system may provide a warning, such as in the form of sound, visual warning, vibration, and / or a rapid brake pulse.

[0185] In at least one embodiment, the AEB system detects an imminent forward collision with another vehicle or other object and may automatically apply the brakes if the driver does not take corrective action within specified time or distance parameters. In at least one embodiment, the AEB system may use front camera(s) and / or RADAR sensor(s) (1060) coupled to a dedicated processor, DSP, FPGA, and / or ASIC. In at least one embodiment, when the AEB system detects a hazard, it will typically first alert the driver to take corrective action to avoid the collision, and if the driver does not take corrective action, the AEB system may automatically apply the brakes in an effort to prevent or at least mitigate the impact of the predicted collision. In at least one embodiment, the AEB system may include techniques such as dynamic brake assist and / or imminent collision braking.

[0186] In at least one embodiment, the LDW system provides visual, auditory, and / or tactile warnings, such as steering wheel or seat vibrations, to alert the driver when the vehicle (1000) crosses lane markings. In at least one embodiment, the LDW system is not activated when the driver indicates an intentional lane departure, such as by activating a turn signal. In at least one embodiment, the LDW system may use front-facing cameras coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which are electrically coupled to provide driver feedback, such as a display, speaker, and / or vibration component. In at least one embodiment, the LKA system is a variation of the LDW system. In at least one embodiment, the LKA system provides steering input or braking to correct the vehicle (1000) when the vehicle (1000) begins to deviate from its lane.

[0187] In at least one embodiment, the BSW system detects and warns drivers of vehicles in the blind spot of the vehicle. In at least one embodiment, the BSW system may provide visual, auditory, and / or tactile warnings to indicate that it is unsafe to merge into or change lanes. In at least one embodiment, the BSW system may provide additional warnings when the driver uses the turn signal. In at least one embodiment, the BSW system may use rear camera(s) and / or RADAR sensor(s) (1060) coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback, such as a display, speaker, and / or vibration component.

[0188] In at least one embodiment, the RCTW system may provide visual, auditory, and / or tactile notification when an object is detected outside the rear-camera range while the vehicle (1000) is reversing. In at least one embodiment, the RCTW system includes an AEB system that ensures the vehicle brakes are applied to avoid a collision. In at least one embodiment, the RCTW system may use one or more rear RADAR sensor(s) (1060) coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which are electrically coupled to provide driver feedback, such as a display, speaker, and / or vibration component.

[0189] In at least one embodiment, conventional ADAS systems may be susceptible to false positive results that can be annoying and distracting to the driver, but are not typically fatal because conventional ADAS systems alert the driver and allow the driver to determine whether a safety condition truly exists and to act accordingly. In at least one embodiment, the vehicle (1000) itself determines whether to pay attention to results from the main computer or the subcomputer (e.g., the first controller or the second controller among the controllers (1036)) in the event of conflicting results. For example, in at least one embodiment, the ADAS system (1038) may be a backup and / or subcomputer for providing cognitive information to a backup computer rationality module. In at least one embodiment, the backup computer rationality monitor may run various redundant software on hardware components to detect failures in cognitive and dynamic driving tasks. In at least one embodiment, outputs from the ADAS system (1038) may be provided to a supervisory MCU. In at least one embodiment, if outputs from the main computer and outputs from the subcomputer conflict, the supervisory MCU determines a method to coordinate the conflict to ensure safe operation.

[0190] In at least one embodiment, the main computer may be configured to provide a reliability score to the supervisory MCU, indicating the reliability of the main computer in the selected result. In at least one embodiment, if the reliability score exceeds a threshold, the supervisory MCU may follow the direction of the main computer regardless of whether the sub-computer provides conflicting or inconsistent results. In at least one embodiment, if the reliability score does not meet the threshold and the main and sub-computers display different results (e.g., conflicting), the supervisory MCU may mediate between the computers to determine an appropriate result.

[0191] In at least one embodiment, the supervisory MCU may be configured to run a neural network(s) trained and configured to determine conditions under which the subcomputer provides false alarms, based at least partially on outputs from the main computer and outputs from the subcomputer. In at least one embodiment, the neural network(s) in the supervisory MCU may learn when the outputs of the subcomputer can be trusted and when they cannot be trusted. For example, in at least one embodiment, when the subcomputer is a RADAR-based FCW system, the neural network(s) in the supervisory MCU may learn when the FCW system identifies metal objects that are actually harmless, such as drain grates or manhole covers, which trigger an alarm. In at least one embodiment, when the subcomputer is a camera-based LDW system, the neural network in the supervisory MCU may learn to override the LDW when cyclists or pedestrians are present and lane departure is actually the safest operation. In at least one embodiment, the supervising MCU may include at least one of a DLA or a GPU suitable for running neural network(s) with associated memory. In at least one embodiment, the supervising MCU may include a component of an SoC(s) (1004) and / or may be included as such a component.

[0192] In at least one embodiment, the ADAS system (1038) may include a subcomputer that performs ADAS functionality using traditional computer vision rules. In at least one embodiment, the subcomputer may use classical computer vision rules (if-then), and the presence of neural network(s) in the supervisory MCU may improve reliability, safety, and performance. For example, in at least one embodiment, various implementations and intentional non-identity make the entire system more fault-tolerant, particularly against failures caused by software (or software-hardware interface) functionality. For example, in at least one embodiment, if there is a software bug or error in the software running on the main computer and non-identical software code running on the subcomputer provides a consistent overall result, then the supervisory MCU may have greater confidence that the overall result is accurate and that the bug in the software or hardware on the main computer does not cause data errors.

[0193] In at least one embodiment, the output of the ADAS system (1038) may be supplied to the perception block of the main computer and / or the dynamic driving task block of the main computer. For example, in at least one embodiment, if the ADAS system (1038) displays a forward collision warning due to an object immediately ahead, the perception block may use this information when identifying objects. In at least one embodiment, the subcomputer may have its own neural network that is trained as described herein and thus reduces the risk of false positives.

[0194] In at least one embodiment, the vehicle (1000) may further include an infotainment SoC (1030) (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system SoC (1030) may not be an SoC in at least one embodiment and may include two or more individual components, without limitation. In at least one embodiment, the infotainment SoC (1030) may include, without limitation, a combination of hardware and software that can be used to provide audio (e.g., music, personal information terminal, navigation commands, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., LTE, WiFi, etc.), and / or information services (e.g., navigation systems, rear-parking assist, radio data system, vehicle-related information such as fuel level, total distance covered, brake fuel level, oil level, door open / close, air filter information, etc.) to the vehicle (1000). For example, the infotainment SoC (1030) may include radios, disc players, navigation systems, video players, USB and Bluetooth connectivity, vehicle computers, in-car entertainment, WiFi, steering wheel audio controls, hands-free voice control, HUD (heads-up display), HMI display (1034), telematics devices, control panels (e.g., for controlling and / or interacting with various components, features, and / or systems), and / or other components.In at least one embodiment, the infotainment SoC (1030) may be further used to provide information (e.g., visual and / or auditory) to the user(s) of the vehicle (1000), such as information from the ADAS system (1038), autonomous driving information such as planned vehicle operations, trajectories, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0195] In at least one embodiment, the infotainment SoC (1030) may include any amount and type of GPU functionality. In at least one embodiment, the infotainment SoC (1030) may communicate with other devices, systems, and / or components of the vehicle (1000) via the bus (1002). In at least one embodiment, the infotainment SoC (1030) may be coupled to a supervisory MCU, so that in the event of a failure of the main controller(s) (1036) (e.g., the main computer and / or backup computers of the vehicle (1000)), the GPU of the infotainment system may perform some self-driving functionality. In at least one embodiment, the infotainment SoC (1030) may put the vehicle (1000) into a chopper in a safe shutdown mode, as described herein.

[0196] In at least one embodiment, the vehicle (1000) may further include a device cluster (1032) (e.g., a digital dash, an electronic device cluster, a digital device panel, etc.). In at least one embodiment, the device cluster (1032) may include, without limitation, a controller and / or a supercomputer (e.g., an individual controller or a supercomputer). In at least one embodiment, the device cluster (1032) may include, without limitation, a set of any number and combination of devices such as a speedometer, fuel level, oil pressure, tachometer, odometer, direction indicators, gear shift position indicator, seat belt warning light(s), parking brake warning light(s), engine malfunction light(s), supplemental restraint system (e.g., airbag) information, lighting controls, safety system controls, navigation information, etc. In some examples, information may be displayed and / or shared between the infotainment SoC (1030) and the device cluster (1032). In at least one embodiment, the device cluster (1032) may be included as part of the infotainment SoC (1030), or vice versa.

[0197] FIG. 10d is a diagram of a system for communication between the cloud-based server(s) of FIG. 10a and an autonomous vehicle (1000) according to at least one embodiment. In at least one embodiment, the system may include any number and type of vehicles, including server(s) (1078), network(s) (1090), and a vehicle (1000), without limitation. In at least one embodiment, the server(s) (1078) may include, without limitation, a plurality of GPUs (1084(A) to 1084(H)) (collectively referred to herein as GPUs (1084)), PCIe switches (1082(A) to 1082(D)) (collectively referred to herein as PCIe switches (1082)), and / or CPUs (1080(A) to 1080(B)) (collectively referred to herein as CPUs (1080)). In at least one embodiment, GPUs (1084), CPUs (1080), and PCIe switches (1082) may be interconnected with high-speed interconnects, such as NVLink interfaces (1088) and / or PCIe connections (1086) developed by NVIDIA, for example and without limitation. In at least one embodiment, GPUs (1084) are connected via an NVLink and / or NVSwitch SoC, and GPUs (1084) and PCIe switches (1082) are connected via PCIe interconnects. Eight GPUs (1084), two CPUs (1080), and four PCIe switches (1082) are exemplified, but this is not intended to be limiting. In at least one embodiment, each server(s) (1078) may include any number of GPUs (1084), CPUs (1080), and / or PCIe switches (1082) in any combination without limitation. For example, in at least one embodiment, each server(s) (1078) may include 8, 16, 32, and / or more GPUs (1084).

[0198] In at least one embodiment, the server(s) (1078) may receive image data representing images showing unexpected or changed road conditions, such as recently undertaken road work, through the network(s) (1090) and from vehicles. In at least one embodiment, the server(s) (1078) may transmit map information (1094) to the network(s) (1090) and to vehicles, including, without limitation, updated or other types of neural networks (1092) and / or information regarding traffic and road conditions. In at least one embodiment, updates to the map information (1094) may, without limitation, include updates to the HD map (1022), such as information regarding construction sites, potholes, detours, flooding, and / or other obstacles. In at least one embodiment, the neural networks (1092) and / or map information (1094) may be derived from new training and / or experiences represented in data received from any number of vehicles in the environment, and / or at least partially based on training performed in a data center (e.g., using server(s) (1078) and / or other servers).

[0199] In at least one embodiment, server(s) (1078) may be used to train a machine learning model (e.g., a neural network) based at least partially on training data. In at least one embodiment, training data may be generated by vehicles and / or generated in a simulation (e.g., using a game engine). In at least one embodiment, any amount of training data is tagged and / or undergoes other preprocessing (e.g., if the associated neural network benefits from supervised learning). In at least one embodiment, any amount of training data is not tagged and / or preprocessed (e.g., if the associated neural network does not require supervised learning). In at least one embodiment, once the machine learning models are trained, the machine learning models may be used by vehicles (e.g., transmitted to vehicles via network(s) (1090)) and / or the machine learning models may be used by server(s) (1078) to remotely monitor the vehicles.

[0200] In at least one embodiment, the server(s) (1078) may receive data from vehicles and apply the data to modern real-time neural networks for real-time intelligent inference. In at least one embodiment, the server(s) (1078) may include deep-learning supercomputers and / or dedicated AI computers powered by GPU(s) (1084), such as DGX and DGX Station machines developed by NVIDIA. However, in at least one embodiment, the server(s) (1078) may include deep learning infrastructure using CPU-powered data centers.

[0201] In at least one embodiment, the deep-learning infrastructure of the server(s) (1078) may be capable of high-speed, real-time inference and may use its capabilities to evaluate and verify the health of processors, software, and / or associated hardware in the vehicle (1000). For example, in at least one embodiment, the deep-learning infrastructure may receive periodic updates from the vehicle (1000), such as a series of images and / or objects located in the series of images by the vehicle (1000) (e.g., through computer vision and / or other machine learning object classification techniques). In at least one embodiment, the deep-learning infrastructure may run its own neural network to identify objects and compare them with objects identified by the vehicle (1000), and if the results do not match and the deep-learning infrastructure concludes that the AI ​​in the vehicle (1000) is malfunctioning, then the server(s) (1078) may take control of the fail-safe computer of the vehicle (1000), notify the passengers, and send a signal to the vehicle (1000) ordering it to complete a safe parking operation.

[0202] In at least one embodiment, the server(s) (1078) may include GPU(s) (1084) and one or more programmable inference accelerators (e.g., NVIDIA's TensorRT 3 devices). In at least one embodiment, a combination of GPU-based servers and inference acceleration may enable real-time responsiveness. In at least one embodiment, such as when performance is less critical, servers powered by CPUs, FPGAs, and other processors may be used for inference. In at least one embodiment, hardware structure(s) (715) are used to perform one or more embodiments. Details regarding the hardware structure(s) (715) are provided herein together with FIG. 7a and / or FIG. 7b.

[0203] In at least one embodiment, at least one component illustrated or described in connection with FIGS. 10a through 10d is used to implement the techniques and / or functions described in connection with FIGS. 1 through 6. In at least one embodiment, accelerator(s) (1014) are hardware resources used to perform image classification using input data comprising a voxel representation of an environment, at least partially based on a data structure for indicating the voxels to be generated during the conversion to the corresponding voxel representation of the point cloud, as described together with FIG. 1 and as otherwise described in this specification.

[0204] computer systems

[0205] FIG. 11 is a block diagram illustrating an exemplary computer system that may be a system having interconnected devices and components, a system-on-a-chip (SOC), or a combination thereof, formed together with a processor that may include execution units for executing instructions according to at least one embodiment. In at least one embodiment, the computer system (1100) may, without limitation, include components such as a processor (1102) that utilizes execution units that include logic for executing algorithms for processing data according to the present disclosure, as in the embodiments described herein. In at least one embodiment, the computer system (1100) may include processors such as the PENTIUM® processor family available from Intel Corporation of Santa Clara, California, Xeon™, Itanium®, XScale™ and / or StrongARM™, Intel® Core™, or Intel® Nervana™ microprocessors, but other systems (including PCs, engineering workstations, set-top boxes, etc., having other microprocessors) may also be used. In at least one embodiment, the computer system (1100) may run a version of the WINDOWS operating system available from Microsoft Corporation of Redmond, Wash., but other operating systems (e.g., UNIX and Linux), built-in software, and / or graphical user interfaces may also be used.

[0206] The embodiments may be used in other devices such as handheld devices and embedded applications. Some examples of handheld devices include cellular phones, Internet Protocol devices, digital cameras, PDAs (personal digital assistants), and handheld PCs. In at least one embodiment, embedded applications may include a microcontroller, a DSP (digital signal processor), a system-on-chip, network computers ("NetPCs"), set-top boxes, network hubs, WAN (wide area network) switches, or any other system capable of executing one or more commands according to at least one embodiment.

[0207] In at least one embodiment, the computer system (1100) may, without limitation, include a processor (1102) which may, without limitation, include one or more execution units (1108) that perform machine learning model training and / or inference according to the techniques described herein. In at least one embodiment, the computer system (1100) is a single-processor desktop or server system, but in another embodiment, the computer system (1100) may be a multiprocessor system. In at least one embodiment, the processor (1102) may, without limitation, include any other processor device, such as, for example, a "complex instruction set computer" (CISC) microprocessor, a "reduced instruction set computing" (RISC) microprocessor, a "very long instruction word" (VLIW) microprocessor, a processor implementing a combination of instruction sets, or a digital signal processor. In at least one embodiment, the processor (1102) may be coupled to a processor bus (1110) capable of transmitting data signals between the processor (1102) and other components in the computer system (1100).

[0208] In at least one embodiment, the processor (1102) may include, without limitation, an L1 (Level 1) internal cache memory ("cache") (1104). In at least one embodiment, the processor (1102) may have a single internal cache or multiple levels of internal caches. In at least one embodiment, the cache memory may reside outside the processor (1102). Other embodiments may also include a combination of both internal and external caches depending on specific implementation and needs. In at least one embodiment, the register file (1106) may store different types of data in various registers, without limitation, including integer registers, floating-point registers, status registers, and instruction pointer registers.

[0209] In at least one embodiment, an execution unit (1108) including, without limitation, logic for performing integer and floating-point operations also resides in the processor (1102). In at least one embodiment, the processor (1102) may also include a "ucode" (microcode) "ROM" (read-only memory) that stores microcode for specific macro instructions. In at least one embodiment, the execution unit (1108) may include logic for handling a packed instruction set (1109). In at least one embodiment, by including a packed instruction set (1109) in the instruction set of a general-purpose processor along with associated circuitry for executing instructions, operations used by many multimedia applications can be performed in the processor (1102) using packed data. In at least one embodiment, many multimedia applications can be accelerated and executed more efficiently by using the full width of the processor's data bus to perform operations on packed data, which can eliminate the need to transmit smaller units of data across the processor's data bus to perform one or more operations on one data element at a time.

[0210] In at least one embodiment, the execution unit (1108) may also be used in microcontrollers, embedded processors, graphics devices, DSPs, and other types of logic circuits. In at least one embodiment, the computer system (1100) may include memory (1120), without limitation. In at least one embodiment, the memory (1120) may be a "DRAM" (Dynamic Random Access Memory) device, a "SRAM" (Static Random Access Memory) device, a flash memory device, or other memory device. In at least one embodiment, the memory (1120) may store instruction(s) (1119) and / or data (1121) represented by data signals that can be executed by the processor (1102).

[0211] In at least one embodiment, the system logic chip may be coupled to the processor bus (1110) and the memory (1120). In at least one embodiment, the system logic chip may include, without limitation, a "MCH" (memory controller hub) (1116), and the processor (1102) may communicate with the MCH (1116) via the processor bus (1110). In at least one embodiment, the MCH (1116) may provide a high-bandwidth memory path (1118) to the memory (1120) for storing instructions and data, and for storing graphics commands, data, and textures. In at least one embodiment, the MCH (1116) may direct data signals between the processor (1102), the memory (1120), and other components in the computer system (1100), and bridge data signals between the processor bus (1110), the memory (1120), and the system I / O interface (1122). In at least one embodiment, the system logic chip may provide a graphics port for connecting to a graphics controller. In at least one embodiment, the MCH (1116) may be coupled to memory (1120) via a high-bandwidth memory path (1118), and a graphics / video card (1112) may be coupled to the MCH (1116) via an "AGP" (Accelerated Graphics Port) interconnect (1114).

[0212] In at least one embodiment, the computer system (1100) may use the system I / O interface (1122) as a proprietary hub interface bus to connect the MCH (1116) to an I / O controller hub ("ICH") (1130). In at least one embodiment, the ICH (1130) may provide direct connections to some I / O devices via a local I / O bus. In at least one embodiment, the local I / O bus may include, without limitation, a high-speed I / O bus for connecting peripherals to memory (1120), a chipset, and a processor (1102). Examples may include, without limitation, an audio controller (1129), a firmware hub ("Flash BIOS") (1128), a wireless transceiver (1126), a legacy I / O controller (1123) containing data storage (1124), user input and keyboard interfaces (1125), a serial expansion port (1127), such as a "USB" (Universal Serial Bus) port, and a network controller (1134). In at least one embodiment, the data storage (1124) may include a hard disk drive, a floppy disk drive, a CD-ROM device, a flash memory device, or other mass storage device.

[0213] In at least one embodiment, FIG. 11 illustrates a system comprising interconnected hardware devices or "chips," whereas in other embodiments, FIG. 11 may illustrate an exemplary SoC. In at least one embodiment, the devices illustrated in FIG. 11 may be interconnected with proprietary interconnects, standardized interconnects (e.g., PCIe), or some combination thereof. In at least one embodiment, one or more components of a computer system (1100) are interconnected using CXL (compute express link) interconnects.

[0214] Logic (715) is used to perform inference and / or training operations associated with one or more embodiments. Details regarding the logic (715) are provided herein together with FIG. 7a and / or FIG. 7b. In at least one embodiment, the logic (715) may be used in a computer system (1100) to infer or predict operations based at least partially on weight parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0215] In at least one embodiment, at least one component illustrated or described with respect to FIG. 11 is used to implement the techniques and / or functions described in relation to FIGS. 1 through 6. In at least one embodiment, a processor (1102) performs image classification using input data comprising a voxel representation of an environment, at least partially based on a data structure for indicating the voxels to be generated during the conversion to the corresponding voxel representation of the point cloud, as described together with FIG. 1 and as otherwise described in this specification.

[0216] FIG. 12 is a block diagram illustrating an electronic device (1200) for using a processor (1210) according to at least one embodiment. In at least one embodiment, the electronic device (1200) may be, for example and without limitation, a laptop, a tower server, a rack server, a blade server, a laptop, a desktop, a tablet, a mobile device, a telephone, an embedded computer, or any other suitable electronic device.

[0217] In at least one embodiment, the electronic device (1200) may, without limitation, include a processor (1210) that is communicably coupled to any suitable number or type of components, peripherals, modules, or devices. In at least one embodiment, the processor (1210) is I2 They are coupled using a bus or interface such as a C bus, "SMBus" (System Management Bus), LPC (Low Pin Count) bus, "SPI" (Serial Peripheral Interface), "HDA" (High Definition Audio) bus, "SATA" (Serial Advance Technology Attachment) bus, "USB" (Universal Serial Bus) (versions 1, 2, 3, etc.), or "UART" (Universal Asynchronous Receiver / Transmitter) bus. In at least one embodiment, FIG. 12 illustrates a system comprising interconnected hardware devices or "chips," whereas in other embodiments, FIG. 12 may illustrate an exemplary SoC. In at least one embodiment, the devices illustrated in FIG. 12 may be interconnected with proprietary interconnects, standardized interconnects (e.g., PCIe), or some combination thereof. In at least one embodiment, one or more components of FIG. 12 are interconnected using CXL (compute express link) interconnects.

[0218] In at least one embodiment, FIG. 12 includes a display (1224), a touch screen (1225), a touch pad (1230), an NFC (Near Field Communications unit) (1245), a sensor hub (1240), a thermal sensor (1246), an EC (Express Chipset) (1235), a TPM (Trusted Platform Module) (1238), a BIOS, a FW flash (BIOS / firmware / flash memory) (1222), a DSP (1260), a drive (1220) such as an SSD (Solid State Disk) or HDD (Hard Disk Drive), a WLAN (wireless local area network unit) (1250), a Bluetooth unit (1252), a WWAN (Wireless Wide Area Network unit) (1256), a GPS (Global Positioning System) unit (1255), a USB 3.0 camera (camera) (1254) such as a USB 3.0 camera, and / or, for example, It may include an "LPDDR3" (Low Power Double Data Rate ("LPDDR")) memory unit (1215) implemented according to the LPDDR3 standard. Each of these components may be implemented in any suitable manner.

[0219] In at least one embodiment, other components may be communicably coupled to the processor (1210) through the components described herein. In at least one embodiment, an accelerometer (1241), an "ALS" (ambient light sensor) (1242), a compass (1243), and a gyroscope (1244) may be communicably coupled to a sensor hub (1240). In at least one embodiment, a heat sensor (1239), a fan (1237), a keyboard (1236), and a touchpad (1230) may be communicably coupled to an EC (1235). In at least one embodiment, speakers (1263), headphones (1264), and a "mic" (microphone) (1265) may be communicably coupled to an audio unit ("audio codec and class D amplifier") (1262), which may eventually be communicably coupled to a DSP (1260). In at least one embodiment, the audio unit (1262) may include an audio coder / decoder (“codec”) and a Class D amplifier, for example and without limitation. In at least one embodiment, a SIM card (“SIM”) (1257) may be communicably coupled to the WWAN unit (1256). In at least one embodiment, components such as the WWAN unit (1256), as well as the WLAN unit (1250) and the Bluetooth unit (1252), may be implemented as “NGFF” (Next Generation Form Factor).

[0220] Logic (715) is used to perform inference and / or training operations associated with one or more embodiments. Details regarding the logic (715) are provided herein together with FIG. 7a and / or FIG. 7b. In at least one embodiment, the logic (715) may be used in an electronic device (1200) to infer or predict operations based at least partially on weight parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0221] In at least one embodiment, at least one component illustrated or described with respect to FIG. 12 is used to implement the techniques and / or functions described in relation to FIGS. 1 through 6. In at least one embodiment, a processor (1210) performs image classification using input data comprising a voxel representation of an environment, at least partially based on a data structure for indicating the voxels to be generated during the conversion to the corresponding voxel representation of the point cloud, as described together with FIG. 1 and as otherwise described in this specification.

[0222] FIG. 13 illustrates a computer system (1300) according to at least one embodiment. In at least one embodiment, the computer system (1300) is configured to implement various processes and methods described throughout the present disclosure.

[0223] In at least one embodiment, the computer system (1300) includes, without limitation, at least one "CPU" (central processing unit) (1302) connected to a communication bus (1310) implemented using any suitable protocol, such as PCI ("Peripheral Component Interconnect"), "PCI-Express" (peripheral component interconnect express), AGP ("Accelerated Graphics Port"), HyperTransport, or any other bus or point-to-point communication protocol(s). In at least one embodiment, the computer system (1300) includes, without limitation, main memory (1304) and control logic (e.g., implemented as hardware, software, or a combination thereof), and data is stored in the main memory (1304), which may take the form of "RAM" (random access memory). In at least one embodiment, a network interface subsystem ("network interface") (1322) provides an interface to other computing devices and networks for receiving data from other systems to the computer system (1300) and transmitting data to other systems.

[0224] In at least one embodiment, the computer system (1300) comprises, without limitation, input devices (1308), a parallel processing system (1312), and display devices (1306) that can be implemented using a conventional CRT (cathode ray tube), LCD (liquid crystal display), LED (light emitting diode) display, plasma display, or other suitable display technology. In at least one embodiment, user input is received from input devices (1308), such as a keyboard, mouse, touchpad, microphone, etc. In at least one embodiment, each module described herein may be placed on a single semiconductor platform to form a processing system.

[0225] The logic (715) is used to perform inference and / or training operations associated with one or more embodiments. Details regarding the inference and / or training logic (715) are provided herein together with FIG. 7a and / or FIG. 7b. In at least one embodiment, the logic (715) may be used in a computer system (1300) to infer or predict operations based at least partially on weight parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0226] In at least one embodiment, at least one component illustrated or described with respect to FIG. 13 is used to implement the techniques and / or functions described in relation to FIGS. 1 through 6. In at least one embodiment, a computer system (1300) performs one or more operations of image classification using input data comprising a voxel representation of an environment based at least partially on a data structure to indicate the voxels to be generated during the conversion to the corresponding voxel representation of the point cloud, as described together with FIG. 1 and as otherwise described in this specification.

[0227] FIG. 14 illustrates a computer system (1400) according to at least one embodiment. In at least one embodiment, the computer system (1400) includes, without limitation, a computer (1410) and a USB stick (1420). In at least one embodiment, the computer (1410) may include, without limitation, any number and type of processor(s) (not shown) and memory (not shown). In at least one embodiment, the computer (1410) includes, without limitation, a server, a cloud instance, a laptop, and a desktop computer.

[0228] In at least one embodiment, the USB stick (1420) comprises, without limitation, a processing unit (1430), a USB interface (1440), and USB interface logic (1450). In at least one embodiment, the processing unit (1430) may be any instruction execution system, device, or device capable of executing instructions. In at least one embodiment, the processing unit (1430) may comprise, without limitation, any number and type of processing cores (not shown). In at least one embodiment, the processing unit (1430) comprises an "ASIC" (application specific integrated circuit) optimized to perform any quantity and type of operations associated with machine learning. For example, in at least one embodiment, the processing unit (1430) is a "TPC" (tensor processing unit) optimized to perform machine learning inference operations. In at least one embodiment, the processing unit (1430) is a "VPU" (vision processing unit) optimized to perform machine vision and machine learning inference operations.

[0229] In at least one embodiment, the USB interface (1440) may be any type of USB connector or USB socket. For example, in at least one embodiment, the USB interface (1440) is a USB 3.0 Type-C socket for data and power. In at least one embodiment, the USB interface (1440) is a USB 3.0 Type-A connector. In at least one embodiment, the USB interface logic (1450) may include any amount and type of logic that enables the processing unit (1430) to interface with devices (e.g., a computer (1410)) through the USB connector (1440).

[0230] Logic (715) is used to perform inference and / or training operations associated with one or more embodiments. Details regarding the logic (715) are provided herein together with FIG. 7a and / or FIG. 7b. In at least one embodiment, the logic (715) may be used in a computer system (1400) to infer or predict operations based at least partially on weight parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0231] In at least one embodiment, at least one component illustrated or described with respect to FIG. 14 is used to implement the techniques and / or functions described in relation to FIGS. 1 through 6. In at least one embodiment, a computer system (1400) performs one or more operations of image classification using input data comprising a voxel representation of an environment, at least partially based on a data structure for indicating the voxels to be generated during the conversion to the corresponding voxel representation of a point cloud, as described together with FIG. 1 and as otherwise described in this specification.

[0232] FIG. 15a illustrates an exemplary architecture in which a plurality of GPUs (1510(1)-1510(N)) are communicably coupled to a plurality of multi-core processors (1505(1)-1505(M)) via high-speed links (1540(1)-1540(N)) (e.g., buses, point-to-point interconnects, etc.). In at least one embodiment, the high-speed links (1540(1)-1540(N)) support communication throughputs of 4 GB / s, 30 GB / s, 80 GB / s or more. In at least one embodiment, various interconnect protocols including, but not limited to, PCIe 4.0 or 5.0 and NVLink 2.0 may be used. In various drawings, "N" and "M" represent positive integers, and their values ​​may vary from drawing to drawing. In at least one embodiment, one or more GPUs within a plurality of GPUs (1510(1)-1510(N)) comprise one or more graphics cores (also simply referred to as “cores”) (1800) as disclosed in FIG. 18a and FIG. 18b. In at least one embodiment, one or more graphics cores (1800) may be referred to as streaming multiprocessors (“SMs”), stream processors (“SPs”), stream processing units (“SPUs”), computing units (“CUs”), execution units (“EUs”), and / or slices, wherein a slice in this context may refer to a portion of processing resources in a processing unit (e.g., 16 cores, ray tracing unit, thread director, or scheduler).

[0233] Additionally, and in at least one embodiment, two or more of the GPUs (1510) are interconnected via high-speed links (1529(1)-1529(2)), which may be implemented using protocols / links similar to or different from those used for the high-speed links (1540(1)-1540(N)). Similarly, two or more of the multi-core processors (1505) may be connected via high-speed links (1528), which may be symmetric multiprocessor (SMP) buses operating at 20 GB / s, 30 GB / s, 120 GB / s, or higher. Alternatively, all communication between the various system components illustrated in FIG. 15a may be achieved using similar protocols / links (e.g., through a common interconnection fabric).

[0234] In at least one embodiment, each multi-core processor (1505) is each communically coupled to processor memory (1501(1)-1501(M)) through memory interconnects (1526(1)-1526(M)), and each GPU (1510(1)-1510(N)) is each communically coupled to GPU memory (1520(1)-1520(N)) through GPU memory interconnects (1550(1)-1550(N)). In at least one embodiment, the memory interconnects (1526 and 1550) may utilize similar or different memory access technologies. By way of being not limited to, for example, processor memories (1501(1)-1501(M)) and GPU memories (1520) may be volatile memories such as DRAMs (dynamic random access memories) (including stacked DRAMs), GDDR (Graphics DDR SDRAM) (e.g., GDDR5, GDDR6), or HBM (High Bandwidth Memory) and / or non-volatile memories such as 3D XPoint or Nano-Ram. In at least one embodiment, some parts of the processor memories (1501) may be volatile memories, and other parts may be non-volatile memories (e.g., using a 2LM (two-level memory) layer).

[0235] As described in this specification, various multi-core processors (1505) and GPUs (1510) may each be physically coupled to specific memories (1501, 1520), and / or, an integrated memory architecture may be implemented in which a virtual system address space (also referred to as an "effective address" space) is distributed among various physical memories. For example, processor memories (1501(1)-1501(M)) may each contain a system memory address space of 64 GB, and GPU memories (1520(1)-1520(N)) may each contain a system memory address space of 32 GB, thus resulting in a total addressable memory of 256 GB when M=2 and N=4. Other values ​​for N and M are possible.

[0236] FIG. 15b illustrates additional details regarding the interconnection between a multi-core processor (1507) and a graphics acceleration module (1546) according to one exemplary embodiment. In at least one embodiment, the graphics acceleration module (1546) may include one or more GPU chips integrated on a line card coupled to the processor (1507) via a high-speed link (1540) (e.g., PCIe bus, NVLink, etc.). In at least one embodiment, the graphics acceleration module (1546) may alternatively be integrated on a package or chip with the processor (1507).

[0237] In at least one embodiment, the processor (1507) comprises a plurality of cores (1560A-1560D) (which may be referred to as “execution units”), each having a translation index buffer (“TLB”) (1561A-1561D) and one or more caches (1562A-1562D). In at least one embodiment, the cores (1560A-1560D) may include various other components for executing instructions not illustrated and processing data. In at least one embodiment, the caches (1562A-1562D) may include Level 1 (L1) and Level 2 (L2) caches. Additionally, one or more shared caches (1556) may be included in the caches (1562A-1562D) and may be shared by sets of cores (1560A-1560D). For example, one embodiment of the processor (1507) includes 24 cores, each having its own L1 cache, 12 shared L2 caches, and 12 shared L3 caches. In this embodiment, one or more L2 and L3 caches are shared by two adjacent cores. In at least one embodiment, the processor (1507) and the graphics acceleration module (1546) are connected to a system memory (1514) which may include the processor memories (1501(1)-1501(M)) of FIG. 15a.

[0238] In at least one embodiment, coherence is maintained for data and instructions stored in various caches (1562A-1562D, 1556) and system memory (1514) through inter-core communication via the coherence bus (1564). In at least one embodiment, for example, each cache may have associated cache coherence logic / circuit to communicate via the coherence bus (1564) in response to detected reads or writes for specific cache lines. In at least one embodiment, a cache snooping protocol is implemented via the coherence bus (1564) to snoop cache accesses.

[0239] In at least one embodiment, the proxy circuit (1525) communicateably couples the graphics acceleration module (1546) to the coherence bus (1564), allowing the graphics acceleration module (1546) to participate in the cache coherence protocol as a peer of the cores (1560A-1560D). In particular, in at least one embodiment, the interface (1535) provides connectivity to the proxy circuit (1525) via the high-speed link (1540), and the interface (1537) connects the graphics acceleration module (1546) to the high-speed link (1540).

[0240] In at least one embodiment, the accelerator integration circuit (1536) provides cache management, memory access, context management, and interrupt management services on behalf of the plurality of graphics processing engines (1531(1)-1531(N)) of the graphics acceleration module (1546). In at least one embodiment, the graphics processing engines (1531(1)-1531(N)) may each include a separate graphics processing unit (GPU). In at least one embodiment, the plurality of graphics processing engines (1531(1)-1531(N)) of the graphics acceleration module (1546) include one or more graphics cores (1800) as discussed in relation to FIG. 18a and FIG. 18b. In at least one embodiment, the graphics processing engines (1531(1)-1531(N)) may alternatively include different types of graphics processing engines within the GPU, such as graphics execution units, media processing engines (e.g., video encoders / decoders), samplers, and bullet engines. In at least one embodiment, the graphics acceleration module (1546) may be a GPU having a plurality of graphics processing engines (1531(1)-1531(N)), or the graphics processing engines (1531(1)-1531(N)) may be discrete GPUs integrated on a common package, line card, or chip.

[0241] In at least one embodiment, the accelerator integrated circuit (1536) includes a memory management unit (MMU) (1539) for performing various memory management functions, such as virtual-to-physical memory translations (also referred to as effective-to-actual memory translations) and memory access protocols for accessing system memory (1514). In at least one embodiment, the MMU (1539) may also include a translation lookaside buffer (TLB) (not shown) for caching virtual / effective to physical / actual address translations. In at least one embodiment, the cache (1538) may store commands and data for efficient access by graphics processing engines (1531(1)-1531(N)). In at least one embodiment, data stored in the cache (1538) and graphics memories (1533(1)-1533(M)) is maintained coherently with the core caches (1562A-1562D, 1556) and system memory (1514), possibly using a fetch unit (1544). As mentioned, this can be achieved through a proxy circuit (1525) on behalf of the cache (1538) and memories (1533(1)-1533(M)) (e.g., sending updates related to modifications / accesses of cache lines on the processor caches (1562A-1562D, 1556) to the cache (1538) and receiving updates from the cache (1538)).

[0242] In at least one embodiment, a set of registers (1545) stores context data for threads executed by graphics processing engines (1531(1)-1531(N)), and a context management circuit (1548) manages thread contexts. For example, the context management circuit (1548) may perform save and restore operations to save and restore the contexts of various threads during context switches (e.g., a first thread is saved and a second thread is saved so that the second thread can be executed by the graphics processing engine). For example, during a context switch, the context management circuit (1548) may save the current register values ​​to a designated area in memory (e.g., identified by a context pointer). Next, the register values ​​may be restored when returning to the context. In at least one embodiment, an interrupt management circuit (1547) receives and processes interrupts received from a system device.

[0243] In at least one embodiment, virtual / effective addresses from the graphics processing engine (1531) are converted by the MMU (1539) into actual / physical addresses in system memory (1514). In at least one embodiment, the accelerator integration circuit (1536) supports a number of (e.g., 4, 8, 16) graphics accelerator modules (1546) and / or other accelerator devices. In at least one embodiment, a graphics accelerator module (1546) may be dedicated to a single application running on the processor (1507) or may be shared among a number of applications. In at least one embodiment, a virtualized graphics execution environment is presented in which the resources of the graphics processing engines (1531(1)-1531(N)) are shared with a number of applications or virtual machines (VMs). In at least one embodiment, the resources may be subdivided into "slices" that are allocated to different VMs and / or applications based on processing requirements and priorities associated with the VMs and / or applications.

[0244] In at least one embodiment, the accelerator integration circuit (1536) acts as a bridge to the system for the graphics acceleration module (1546) and provides address translation and system memory cache services. Additionally, in at least one embodiment, the accelerator integration circuit (1536) may provide virtualization facilities for the host processor to manage the virtualization, interrupts, and memory management of the graphics processing engines (1531(1)-1531(N)).

[0245] In at least one embodiment, since the hardware resources of the graphics processing engines (1531(1)-1531(N)) are explicitly mapped to the actual address space shown by the host processor (1507), any host processor can directly address these resources using an effective address value. In at least one embodiment, one function of the accelerator integration circuit (1536) is the physical separation of the graphics processing engines (1531(1)-1531(N)), and thus, they appear in the system as independent units.

[0246] In at least one embodiment, one or more graphics memories (1533(1)-1533(M)) are each coupled to each of the graphics processing engines (1531(1)-1531(N)), and N=M. In at least one embodiment, the graphics memories (1533(1)-1533(M)) store instructions and data processed by each of the graphics processing engines (1531(1)-1531(N)). In at least one embodiment, the graphics memories (1533(1)-1533(M)) may be volatile memories such as DRAMs (including stacked DRAMs), GDDR memory (e.g., GDDR5, GDDR6), or HBM, and / or non-volatile memories such as 3D XPoint or Nano-Ram.

[0247] In at least one embodiment, to reduce data traffic through the high-speed link (1540), biasing techniques may be used to ensure that the data stored in the graphics memories (1533(1)-1533(M)) is data that is most frequently used by the graphics processing engines (1531(1)-1531(N)) and preferably not used (at least not frequently) by the cores (1560A-1560D). Similarly, in at least one embodiment, a biasing mechanism attempts to retain data required by the cores (and preferably not the graphics processing engines (1531(1)-1531(N))) within the caches (1562A-1562D, 1556) and system memory (1514).

[0248] FIG. 15c illustrates another exemplary embodiment in which an accelerator integrated circuit (1536) is integrated within a processor (1507). In this embodiment, graphics processing engines (1531(1)-1531(N)) communicate directly with the accelerator integrated circuit (1536) via a high-speed link (1540) through an interface (1537) and an interface (1535) (again, which may be any form of bus or interface protocol). In at least one embodiment, the accelerator integrated circuit (1536) can perform operations similar to those described in relation to FIG. 15b, but can potentially perform them at a higher throughput if given close proximity to the coherence bus (1564) and caches (1562A-1562D, 1556). In at least one embodiment, the accelerator integration circuit supports different programming models, including a dedicated-process programming model (without graphics acceleration module virtualization) and shared programming models (with virtualization), which may include programming models controlled by the accelerator integration circuit (1536) and programming models controlled by the graphics acceleration module (1546).

[0249] In at least one embodiment, the graphics processing engines (1531(1)-1531(N)) are dedicated to a single application or process under a single operating system. In at least one embodiment, a single application can funnel other application requests to the graphics processing engines (1531(1)-1531(N)), thus providing virtualization within a VM / partition.

[0250] In at least one embodiment, the graphics processing engines (1531(1)-1531(N)) may be shared by multiple VM / application partitions. In at least one embodiment, the sharing models may use a system hypervisor to virtualize the graphics processing engines (1531(1)-1531(N)) to allow access by each operating system. In at least one embodiment, for a single-partition system without a hypervisor, the graphics processing engines (1531(1)-1531(N)) are owned by the operating system. In at least one embodiment, the operating system may virtualize the graphics processing engines (1531(1)-1531(N)) to provide access to each process or application.

[0251] In at least one embodiment, a graphics acceleration module (1546) or an individual graphics processing engine (1531(1)-1531(N)) selects a process element using a process handle. In at least one embodiment, process elements are stored in system memory (1514) and are addressable using the effective address-to-actual address translation technique described herein. In at least one embodiment, the process handle may be an implementation-specific value provided to the host process when registering its context with the graphics processing engine (1531(1)-1531(N)) (i.e., calling system software to add a process element to a process element linked list). In at least one embodiment, the lower 16 bits of the process handle may be an offset of a process element within a process element linked list.

[0252] FIG. 15d illustrates an exemplary accelerator integration slice (1590). In at least one embodiment, the “slice” comprises a specified portion of the processing resources of the accelerator integration circuit (1536). In at least one embodiment, the application stores process elements (1583) in an effective address space (1582) within system memory (1514). In at least one embodiment, the process elements (1583) are stored in response to GPU calls (1581) from applications (1580) running on the processor (1507). In at least one embodiment, the process element (1583) contains a process state for the corresponding application (1580). In at least one embodiment, the work descriptor (WD) (1584) contained in the process element (1583) may be a single task requested by the application or may contain a pointer to a queue of tasks. In at least one embodiment, WD (1584) is a pointer to a job request queue in the application's effective address space (1582).

[0253] In at least one embodiment, the graphics acceleration module (1546) and / or individual graphics processing engines (1531(1)-1531(N)) may be shared by all processes in the system or a subset thereof. In at least one embodiment, infrastructure may be included for setting up process states to start work in a virtualized environment and transmitting WD (1584) to the graphics acceleration module (1546).

[0254] In at least one embodiment, the dedicated process programming model is implementation-specific. In at least one embodiment, in this model, a single process owns a graphics acceleration module (1546) or a separate graphics processing engine (1531). In at least one embodiment, when the graphics acceleration module (1546) is owned by a single process, the hypervisor initializes the accelerator integration circuit (1536) for the owning partition, and the operating system initializes the accelerator integration circuit (1536) for the owning process when the graphics acceleration module (1546) is allocated.

[0255] In at least one embodiment, during operation, the WD fetch unit (1591) in the accelerator integration slice (1590) fetches the next WD (1584), which includes an indication of the operation to be performed by one or more graphics processing engines of the graphics acceleration module (1546). In at least one embodiment, data from the WD (1584) may be stored in registers (1545) and may be used by the MMU (1539), interrupt management circuit (1547), and / or context management circuit (1548) as illustrated. For example, one embodiment of the MMU (1539) includes a segment / page work circuit for accessing segment / page tables (1586) within the OS virtual address space (1585). In at least one embodiment, the interrupt management circuit (1547) may process interrupt events (1592) received from the graphics acceleration module (1546). In at least one embodiment, when performing graphic operations, the effective address (1593) generated by the graphic processing engine (1531(1)-1531(N)) is converted into an actual address by the MMU (1539).

[0256] In at least one embodiment, the registers (1545) are duplicated for each graphics processing engine (1531(1)-1531(N)) and / or graphics acceleration module (1546) and may be initialized by a hypervisor or operating system. In at least one embodiment, each of these duplicated registers may be included in an accelerator integration slice (1590). Exemplary registers that may be initialized by a hypervisor are shown in Table 1.

[0257]

[0258] Exemplary registers that can be initialized by the operating system are shown in Table 2.

[0259]

[0260] In at least one embodiment, each WD (1584) is specific to a specific graphics acceleration module (1546) and / or graphics processing engines (1531(1)-1531(N)). In at least one embodiment, it may contain all information required by the graphics processing engine (1531(1)-1531(N)) to perform a task, or it may be a pointer to a memory location that has set up a command queue for the task to be completed by the application.

[0261] FIG. 15e illustrates additional details regarding one exemplary embodiment of a shared model. This embodiment includes a hypervisor physical address space (1598) in which a process element list (1599) is stored. In at least one embodiment, the hypervisor physical address space (1598) is accessible through a hypervisor (1596) that virtualizes graphics acceleration module engines for an operating system (1595).

[0262] In at least one embodiment, the shared programming model allows all partitions in the system or all processes from a subset thereof or a subset thereof to use the graphics acceleration module (1546). In at least one embodiment, there are two programming models in which the graphics acceleration module (1546) is shared by a number of processes and partitions, namely time-sliced ​​shared and graphics-oriented shared.

[0263] In at least one embodiment, in this model, the system hypervisor (1596) owns the graphics acceleration module (1546) and makes its functions available to all operating systems (1595). In at least one embodiment, for the graphics acceleration module (1546) to support virtualization by the system hypervisor (1596), the graphics acceleration module (1546) may adhere to certain requirements, such as (1) the application's task request must be autonomous (i.e., no state needs to be maintained between tasks), or the graphics acceleration module (1546) must provide a context save and restore mechanism; (2) the application's task request must be guaranteed by the graphics acceleration module (1546) to be completed within a specified amount of time including any transformation failures, or the graphics acceleration module (1546) must provide the ability to preempt the processing of tasks; and (3) the graphics acceleration module (1546) must be guaranteed fairness between processes when operating in a directional shared programming model.

[0264] In at least one embodiment, the application (1580) is required to make an operating system (1595) system call with a graphics acceleration module type, a work descriptor (WD), an authority mask register (AMR) value, and a context save / restore area pointer (CSRP). In at least one embodiment, the graphics acceleration module type describes a targeted acceleration function for the system call. In at least one embodiment, the graphics acceleration module type may be a system-specific value. In at least one embodiment, the WD may be in the form of a graphics acceleration module (1546) command, a valid address pointer for a user-defined structure, a valid address pointer for a queue of commands, or any other data structure describing the work to be performed by the graphics acceleration module (1546).

[0265] In at least one embodiment, the AMR value is the AMR state to be used for the current process. In at least one embodiment, the value passed to the operating system is similar to the application that sets the AMR. In at least one embodiment, if the accelerator integration circuit (1536) (not shown) and the graphics acceleration module (1546) implementations do not support the User Authority Mask Override Register (UAMOR), the operating system may apply the current UAMOR value to the AMR value before passing the AMR in a hypervisor call. In at least one embodiment, the hypervisor (1596) may optionally apply the current Authority Mask Override Register (AMOR) value before placing the AMR in the process element (1583). In at least one embodiment, the CSRP is one of the registers (1545) containing the valid address of a region in the application's valid address space (1582) for the graphics acceleration module (1546) to save and restore the context state. In at least one embodiment, this pointer is optional when no state needs to be saved between tasks or when a task is preempted. In at least one embodiment, the context save / restore area may be a fixed system memory.

[0266] When receiving a system call, the operating system (1595) can verify that the application (1580) is registered and authorized to use the graphics acceleration module (1546). In at least one embodiment, the operating system (1595) then calls the hypervisor (1596) with the information shown in Table 3.

[0267]

[0268] In at least one embodiment, upon receiving a hypervisor call, the hypervisor (1596) verifies that the operating system (1595) is registered and authorized to use the graphics acceleration module (1546). In at least one embodiment, the hypervisor (1596) then adds the process element (1583) to the process element linked list for the corresponding graphics acceleration module (1546) type. In at least one embodiment, the process element may include the information shown in Table 4.

[0269]

[0270] In at least one embodiment, the hypervisor initializes a plurality of accelerator integration slice (1590) registers (1545).

[0271] As illustrated in FIG. 15f, in at least one embodiment, an integrated memory addressable through a common virtual memory address space used to access physical processor memories (1501(1)-1501(N)) and GPU memories (1520(1)-1520(N)) is used. In this implementation, operations executed on GPUs (1510(1)-1510(N)) use the same virtual / effective memory address space to access processor memories (1501(1)-1501(M)), and vice versa, thereby simplifying programmability. In at least one embodiment, a first portion of the virtual / effective address space is allocated to processor memory (1501(1)), a second portion is allocated to second processor memory (1501(N)), a third portion is allocated to GPU memory (1520(1)), and so on. In at least one embodiment, the entire virtual / effective memory space (sometimes referred to as the effective address space) is distributed across the processor memories (1501) and GPU memories (1520), respectively, thereby allowing any processor or GPU to access any physical memory having a virtual address mapped to that memory.

[0272] In at least one embodiment, bias / coherence management circuits (1594A-1594E) within one or more of the MMUs (1539A-1539E) ensure cache coherence between the caches of one or more host processors (e.g., 1505) and GPUs (1510) and implement biasing techniques that indicate physical memories where specific types of data are to be stored. In at least one embodiment, multiple instances of the bias / coherence management circuits (1594A-1594E) are illustrated in FIG. 15f, but the bias / coherence circuits may be implemented within the MMU of one or more host processors (1505) and / or within the accelerator integration circuit (1536).

[0273] One embodiment allows GPU memories (1520) to be mapped as part of system memory and accessed using shared virtual memory (SVM) technology without suffering from performance disadvantages associated with full system cache coherence. In at least one embodiment, the ability for GPU memories (1520) to be accessed as system memory without burdensome cache coherence overhead provides a beneficial operating environment for GPU offload. In at least one embodiment, this configuration allows software on the host processor (1505) to set up operands and access computing results without the overhead of traditional I / O DMA data copies. In at least one embodiment, these traditional copies involve driver calls, interrupts, and memory-mapped I / O (MMIO) accesses, all of which are inefficient compared to simple memory accesses. In at least one embodiment, the ability to access GPU memories (1520) without cache coherence overhead may be important for the execution time of offloaded computing. In at least one embodiment, for example, in the case of substantial streaming write memory traffic, cache coherence overhead can significantly reduce the effective write bandwidth shown by the GPU (1510). In at least one embodiment, the efficiency of operand setup, the efficiency of result access, and the efficiency of GPU computing may play a role in determining the validity of GPU offload.

[0274] In at least one embodiment, the selection of GPU bias and host processor bias is driven by a bias tracker data structure. In at least one embodiment, a bias table may be used, for example, a page-grained structure (e.g., controlled by the grain of memory pages) containing 1 or 2 bits per GPU-attached memory page. In at least one embodiment, the bias table may be implemented in a stolen memory range of one or more GPU memories (1520), with or without a bias cache in the GPU (1510), for example, to cache frequently / recently used entries of the bias table. Alternatively, in at least one embodiment, the entire bias table may be maintained within the GPU.

[0275] In at least one embodiment, a bias table entry associated with each access to the GPU-attached memory (1520) is accessed prior to the actual access to the GPU memory, thereby causing the following operations. In at least one embodiment, local requests from the GPU (1510) to find their pages in GPU bias are delivered directly to the corresponding GPU memory (1520). In at least one embodiment, local requests from the GPU to find their pages in host bias are delivered to the processor (1505) (e.g., via a high-speed link as described herein). In at least one embodiment, requests from the processor (1505) to find the requested pages in host processor bias complete a request such as a normal memory read. Alternatively, requests directed to GPU-biased pages may be delivered to the GPU (1510). In at least one embodiment, the GPU may then transfer the pages to host processor bias if the pages are not currently in use. In at least one embodiment, the bias state of the page may be changed by a software-based mechanism, a hardware-assisted software-based mechanism, or, for a limited set of cases, by a purely hardware-based mechanism.

[0276] In at least one embodiment, a mechanism for changing the bias state uses an API call (e.g., OpenCL), which ultimately calls the GPU device driver, which ultimately sends a message (or encues a command descriptor) instructing the GPU to change the bias state and, for some transitions, to perform a cache flushing operation on the host. In at least one embodiment, the cache flushing operation is used for a transition from the host processor (1505) bias to the GPU bias, but not for the reverse transition.

[0277] In at least one embodiment, cache coherence is maintained by temporarily rendering GPU-biased pages that cannot be cached by the host processor (1505). In at least one embodiment, to access these pages, the processor (1505) may request access from the GPU (1510), which may or may not immediately grant access. Thus, in at least one embodiment, to reduce communication between the processor (1505) and the GPU (1510), it is beneficial to ensure that GPU-biased pages are requested by the GPU rather than the host processor (1505), and vice versa.

[0278] Hardware structure(s) (715) are used to carry out one or more embodiments. Details regarding the hardware structure(s) (715) may be provided in conjunction with FIG. 7a and / or FIG. 7b.

[0279] FIG. 16 illustrates exemplary integrated circuits and associated graphics processors that may be manufactured using one or more IP cores according to various embodiments described herein. In addition to those illustrated, other logic and circuits including additional graphics processors / cores, peripheral interface controllers, or general-purpose processor cores may be included in at least one embodiment.

[0280] FIG. 16 is a block diagram illustrating an exemplary system-on-chip integrated circuit (1600) that can be manufactured using one or more IP cores according to at least one embodiment. In at least one embodiment, the integrated circuit (1600) includes one or more application processor(s) (1605) (e.g., CPUs), at least one graphics processor (1610), and may additionally include an image processor (1615) and / or a video processor (1620), any of which may be a modular IP core. In at least one embodiment, the integrated circuit (1600) includes a USB controller (1625), a UART controller (1630), an SPI / SDIO controller (1635), and I 2 2S / I 2 It includes peripheral or bus logic including a 2C controller (1640). In at least one embodiment, the integrated circuit (1600) may include a display device (1645) coupled to one or more of an HDMI (high-definition multimedia interface) controller (1650) and a MIPI (mobile industry processor interface) display interface (1655). In at least one embodiment, storage may be provided by a flash memory subsystem (1660) including flash memory and a flash memory controller. In at least one embodiment, a memory interface may be provided via a memory controller (1665) for accessing SDRAM or SRAM memory devices. In at least one embodiment, some integrated circuits additionally include an embedded security engine (1670).

[0281] Logic (715) is used to perform inference and / or training operations associated with one or more embodiments. Details regarding the logic (715) are provided herein together with FIG. 7a and / or FIG. 7b. In at least one embodiment, the logic (715) may be used in an integrated circuit (1600) to infer or predict operations based, at least partially, on weight parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0282] In at least one embodiment, at least one component illustrated or described in connection with FIGS. 15a through 16 is used to implement the techniques and / or functions described in connection with FIGS. 1 through 6. In at least one embodiment, an SOC integrated circuit (1600) performs one or more operations of image classification using input data comprising a voxel representation of an environment, at least partially based on a data structure used to indicate the voxels to be generated during the conversion to the corresponding voxel representation of a point cloud, as described together with FIG. 1 and as otherwise described herein.

[0283] FIGS. 17a and 17b illustrate exemplary integrated circuits and associated graphics processors that may be manufactured using one or more IP cores according to various embodiments described herein. In addition to those illustrated, other logic and circuits including additional graphics processors / cores, peripheral interface controllers, or general-purpose processor cores may be included in at least one embodiment.

[0284] FIGS. 17a and FIGS. 17b are block diagrams illustrating exemplary graphics processors for use within an SoC according to embodiments described herein. FIG. 17a illustrates an exemplary graphics processor (1710) of a system-on-chip integrated circuit that can be manufactured using one or more IP cores according to at least one embodiment. FIG. 17b illustrates an additional exemplary graphics processor (1740) of a system-on-chip integrated circuit that can be manufactured using one or more IP cores according to at least one embodiment. In at least one embodiment, the graphics processor (1710) of FIG. 17a is a low-power graphics processor core. In at least one embodiment, the graphics processor (1740) of FIG. 17b is a higher-performance graphics processor core. In at least one embodiment, each of the graphics processors (1710, 1740) may be a variation of the graphics processor (1610) of FIG. 16.

[0285] In at least one embodiment, the graphics processor (1710) includes a vertex processor (1705) and one or more fragment processor(s) (1715A-1715N) (e.g., 1715A, 1715B, 1715C, 1715D, to 1715N-1, and 1715N). In at least one embodiment, the graphics processor (1710) may execute different shader programs through separate logic, such that the vertex processor (1705) is optimized to execute operations for vertex shader programs, while one or more fragment processor(s) (1715A-1715N) execute fragment (e.g., pixel) shading operations for fragment or pixel shader programs. In at least one embodiment, the vertex processor (1705) performs the vertex processing stage of the 3D graphics pipeline and generates primitives and vertex data. In at least one embodiment, fragment processor(s) (1715A-1715N) produce a frame buffer that is displayed on a display device using primitive and vertex data generated by a vertex processor (1705). In at least one embodiment, fragment processor(s) (1715A-1715N) are optimized to execute a fragment shader program provided by the OpenGL API, which can be used to perform operations similar to those of a pixel shader program provided by the Direct 3D API.

[0286] In at least one embodiment, the graphics processor (1710) additionally includes one or more memory management units (MMUs) (1720A-1720B), cache(s) (1725A-1725B), and circuit interconnect(s) (1730A-1730B). In at least one embodiment, one or more MMU(s) (1720A-1720B) provide a virtual-to-physical address mapping for the graphics processor (1710), which includes a vertex processor (1705) and / or fragment processor(s) (1715A-1715N) capable of referencing vertex or image / texture data stored in memory in addition to vertex or image / texture data stored in one or more cache(s) (1725A-1725B). In at least one embodiment, one or more MMU(s) (1720A-1720B) are synchronized with other MMUs in the system, including one or more MMUs associated with one or more application processor(s) (1605) of FIG. 16, an image processor (1615), and / or a video processor (1620), so that each processor (1605-1620) can participate in a shared or integrated virtual memory system. In at least one embodiment, one or more circuit interconnect(s) (1730A-1730B) enable the graphics processor (1710) to interface with other IP cores in the SoC via the internal bus of the SoC or via a direct connection.

[0287] In at least one embodiment, the graphics processor (1740) comprises one or more shader core(s) (1755A-1755N) as illustrated in FIG. 17b (e.g., 1755A, 1755B, 1755C, 1755D, 1755E, 1755F, to 1755N-1, and 1755N), which provides an integrated shader core architecture in which a single core or type of core can execute all types of programmable shader code, including shader program code for implementing vertex shaders, fragment shaders, and / or compute shaders. In at least one embodiment, the number of shader cores may vary. In at least one embodiment, the graphics processor (1740) includes an inter-core task manager (1745) acting as a thread dispatcher that dispatches execution threads to one or more shader cores (1755A-1755N), and a tiling unit (1758) that accelerates tiling operations for tile-based rendering, for example, by utilizing local spatial coherence within the scene or by subdividing rendering operations for the scene in image space to optimize the use of internal caches.

[0288] Logic (715) is used to perform inference and / or training operations associated with one or more embodiments. Details regarding the logic (715) are provided herein together with FIG. 7a and / or FIG. 7b. In at least one embodiment, the logic (715) may be used in a graphics processor (1710 and / or 1740) to infer or predict operations based at least partially on weight parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0289] In at least one embodiment, at least one component illustrated or described in connection with FIG. 17a and FIG. 17b is used to implement the techniques and / or functions described in connection with FIG. 1 through FIG. 6. In at least one embodiment, a graphics processor (1740) performs one or more operations of image classification using input data comprising a voxel representation of an environment based, at least partially, on a data structure to indicate the voxels to be generated during the conversion to the corresponding voxel representation of the point cloud, as described in connection with FIG. 1 and as otherwise described in this specification.

[0290] FIGS. 18a and 18b illustrate additional exemplary graphics processor logic according to embodiments described herein. In at least one embodiment, the components illustrated in FIGS. 18a and 18b and described in connection therewith are integrated into a single system, such as a graphics processing unit (GPU), an SoC, or another type of processor. FIG. 18a illustrates a graphics core (1800) that, in at least one embodiment, may be included within the graphics processor (1610) of FIG. 16 and, in at least one embodiment, may be an integrated shader core (1755A-1755N) as in FIG. 17b illustrates a highly parallel general-purpose graphics processing unit ("GPGPU," which may also be referred to as a "graphics processing unit") (1830) suitable for placement on a multi-chip module in at least one embodiment. In at least one embodiment, the graphics processing unit (1830) is a GPGPU that includes a graphics processor. In at least one embodiment, the integrated circuit (1600) includes, for example, a graphics core (1800) for forming the integrated circuit and / or forming an SoC, and the integrated circuit and / or the SoC perform the operations described herein.

[0291] In at least one embodiment, the graphics core (1800) includes a shared instruction cache (1802), a texture unit (1818), and a cache / shared memory (1820) common to execution resources within the graphics core (1800) (e.g., including L1, L2, L3, end-level cache, or other caches). In at least one embodiment, the graphics core (1800) may include a plurality of slices (1801A-1801N) or partitions for each core, and the graphics processor may include a plurality of instances of the graphics core (1800). In at least one embodiment, each slice (1801A-1801N) refers to the graphics core (1800). In at least one embodiment, the slices (1801A-1801N) have sub-slices that are part of the slices (1801A-1801N). In at least one embodiment, the slices (1801A-1801N) are independent of or dependent on other slices. In at least one embodiment, the slices (1801A-1801N) may include support logic including a local instruction cache (1804A-1804N), a thread scheduler (sequencer) (1806A-1806N), a thread dispatcher (1808A-1808N), and a set of registers (1810A-1810N). In at least one embodiment, the slices (1801A-1801N) may include a set of additional function units (AFUs) (1812A-1812N)), floating-point units (FPUs) (1814A-1814N)), arithmetic logic units (ALUs) (1816A-1816N)), address computational units (ACUs) (1813A-1813N)), double-precision floating-point units (DPFPUs) (1815A-1815N)), and matrix processing units (MPUs) (1817A-1817N)).In at least one embodiment, the MPUs (1817A-1817N) are referred to as matrix engines.

[0292] In at least one embodiment, each slice (1801A-1801N) includes one or more engines for floating-point and integer vector operations and one or more engines for accelerating convolution and matrix operations in AI, machine learning, or large dataset workloads. In at least one embodiment, one or more slices (1801A-1801N) include one or more vector engines for computing vectors (e.g., computing mathematical operations on vectors). In at least one embodiment, the vector engine can compute vector operations in 16-bit floating-point (also referred to as "FP16"), 32-bit floating-point (also referred to as "FP32"), or 64-bit floating-point (also referred to as "FP64"). In at least one embodiment, one or more slices (1801A-1801N) comprise 16 vector engines paired with 16 matrix math units to compute matrix / tensor operations, wherein the vector engines and math units are exposed through matrix extensions. In at least one embodiment, a processing unit slices a designated portion of processing resources for, for example, 16 cores and ray tracing units or 8 cores, a thread scheduler, a thread dispatcher, and additional functional units for the processor. In at least one embodiment, the graphics core (1800) comprises one or more matrix engines for computing matrix operations, for example, when computing tensor operations.

[0293] In at least one embodiment, one or more slices (1801A-1801N) include one or more ray tracing units for computing ray tracing operations (e.g., 16 ray tracing units per slice (1801A-1801N)). In at least one embodiment, the ray tracing units compute ray crossing, triangle intersection, bounding box intersection, or other ray tracing operations.

[0294] In at least one embodiment, one or more slices (1801A-1801N) include a media slice that encodes, decodes, and / or transcodes data; scales and / or formats data; and / or performs video quality operations on video data.

[0295] In at least one embodiment, one or more slices (1801A-1801N) are linked to an L2 cache and memory fabric, link connectors, high-bandwidth memory (HBM) stacks (e.g., HBM2e, HDM3), and a media engine. In at least one embodiment, one or more slices (1801A-1801N) include a plurality of cores (e.g., 16 cores) paired to each core and a plurality of ray tracing units (e.g., 16). In at least one embodiment, one or more slices (1801A-1801N) have one or more L1 caches. In at least one embodiment, one or more slices (1801A-1801N) include one or more vector engines; one or more instruction caches storing instructions; one or more L1 caches for caching data; one or more shared local memory (SLM) storing data corresponding to instructions, e.g.; one or more samplers for sampling data; and one or more ray tracing units for performing ray tracing operations. One or more geometries for performing operations in geometry pipelines and / or applying geometric transformations to vertices or polygons; one or more rasterizers for describing an image in a vector graphics format (e.g., geometry) and converting it into a raster image (e.g., a series of pixels, dots, or lines that, when displayed together, create an image represented by the geometry); one or more hierarchical depth buffers (Hiz) for buffering data; and / or one or more pixel backends. In at least one embodiment, the slice (1801A-1801N) includes a memory fabric, e.g., an L2 cache.

[0296] In at least one embodiment, FPUs (1814A-1814N) can perform single-precision (32-bit) and half-precision (16-bit) floating-point operations, whereas DPFPUs (1815A-1815N) can perform double-precision (64-bit) floating-point operations. In at least one embodiment, ALUs (1816A-1816N) can perform variable-precision integer operations with 8-bit, 16-bit, and 32-bit precision and can be configured for mixed-precision operations. In at least one embodiment, MPUs (1817A-1817N) can also be configured for mixed-precision matrix operations, including half-precision floating-point and 8-bit integer operations. In at least one embodiment, the MPUs (1817-1817N) can perform various matrix operations to accelerate machine learning application frameworks, including enabling support for accelerated GEMM (general matrix to matrix multiplication). In at least one embodiment, the AFUs (1812A-1812N) can perform additional logical operations not supported by floating-point or integer units, including trigonometric operations (e.g., sine, cosine, etc.).

[0297] Logic (715) is used to perform inference and / or training operations associated with one or more embodiments. Details regarding the logic (715) are provided herein together with FIG. 7a and / or FIG. 7b. In at least one embodiment, the logic (715) may be used in a graphics core (1800) to infer or predict operations based, at least partially, on weight parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0298] In at least one embodiment, the graphics core (1800) includes an interconnect and link fabric sublayer attached to a switch and GPU-GPU bridge that enables a plurality of graphics processors (1800) (e.g., 8) to be interlinked to each other without glue across the plurality of graphics processors (1800) via loading / storage units (LSUs), data transfer units, and synchronization semantics. In at least one embodiment, the interconnects include standardized interconnects (e.g., PCIe) or a combination thereof.

[0299] In at least one embodiment, the graphics core (1800) comprises a plurality of tiles. In at least one embodiment, the tile is an individual die or one or more dies, wherein the individual dies may be connected to an interconnect (e.g., an embedded multi-die interconnect bridge (EMIB)). In at least one embodiment, the graphics core (1800) comprises a computing tile, a memory tile (e.g., wherein the memory tile may be exclusively accessed by different tiles or different chipsets, e.g., a Rambo tile), a substrate tile, a base tile, an HMB tile, a link tile, and an EMIB tile, wherein all tiles are packaged together in the graphics core (1800) as part of a GPU. In at least one embodiment, the graphics core (1800) may comprise a plurality of tiles in a single package (also referred to as a "multi-tile package"). In at least one embodiment, the computing tile may have eight graphics cores (1800), an L1 cache; The base tile may have a host interface having PCIe 5.0, HBM2e, MDFI, and EMIB, a link tile having eight links, and eight ports having embedded switches. In at least one embodiment, the tiles are connected by face-to-face (F2F) chip-on-chip bonding through fine pitch, 36-micron, microbumps (e.g., copper fillers). In at least one embodiment, the graphics core (1800) includes a memory fabric containing memory and is a tile accessible by a plurality of tiles. In at least one embodiment, the graphics core (1800) stores, accesses, or loads its own hardware contexts in memory, wherein the hardware context is a set of data loaded from registers before the process is resumed, and the hardware context may indicate the state of the hardware (e.g., the state of the GPU).

[0300] In at least one embodiment, the graphics core (1800) includes a serializer / deserializer (SERDES) circuit that converts a serial data stream into a parallel data stream or converts a parallel data stream into a serial data stream.

[0301] In at least one embodiment, the graphics core (1800) includes a high-speed coherent integrated fabric (GPU-to-GPU), loading / storage units, bulk data transfer and synchronous semantics, and GPUs connected via an embedded switch, wherein the GPU-GPU bridge is controlled by a controller.

[0302] In at least one embodiment, the graphics core (1800) performs an API, wherein the API abstracts the hardware of the graphics core (1800) and accesses libraries with instructions for performing mathematical operations (e.g., a mathematical kernel library), deep neural network operations (e.g., a deep neural network library), vector operations, set communications, thread building blocks, video processing, data analysis library, and / or ray tracing operations.

[0303] FIG. 18b illustrates a GPGPU (1830) that, in at least one embodiment, can be configured to enable highly parallel computing operations to be performed by an array of graphics processing units. In at least one embodiment, the GPGPU (1830) can be directly linked to other instances of the GPGPU (1830) to create a multi-GPU cluster to improve the training speed for deep neural networks. In at least one embodiment, the GPGPU (1830) includes a host interface (1832) that enables a connection with a host processor. In at least one embodiment, the host interface (1832) is a PCI Express interface. In at least one embodiment, the host interface (1832) may be a vendor-specific communication interface or a communication fabric. In at least one embodiment, the GPGPU (1830) receives commands from a host processor and uses a global scheduler (1834) (which may be referred to as a thread sequencer and / or asynchronous computing engine) to distribute execution threads associated with these commands to a set of computing clusters (1836A-1836H). In at least one embodiment, the computing clusters (1836A-1836H) share a cache memory (1838). In at least one embodiment, the cache memory (1838) may serve as an upper-level cache for the cache memories within the computing clusters (1836A-1836H). In at least one embodiment, the computing clusters (1836A-1836H) include slices or are referred to as "slices". In at least one embodiment, the GPGPU (1830) is part of an SoC, such as part of an integrated circuit (1600) (Fig. 16).

[0304] In at least one embodiment, the GPGPU (1830) includes memory (1844A-1844B) combined with computing clusters (1836A-1836H) through a set of memory controllers (1842A-1842B) (e.g., one or more controllers for HBM2e). In at least one embodiment, the memory (1844A-1844B) may include various types of memory devices, such as dynamic random access memory (DRAM) or graphics random access memory, including graphics double data rate (GDDR) memory, synchronous graphics random access memory (SGRAM).

[0305] In at least one embodiment, computing clusters (1836A-1836H) each include a set of graphics cores, such as the graphics core (1800) of FIG. 18a, which may include a plurality of types of integer and floating-point logic units capable of performing computing operations in a range of precisions suitable for machine learning computing. For example, in at least one embodiment, at least a subset of floating-point units in each of computing clusters (1836A-1836H) may be configured to perform 16-bit or 32-bit floating-point operations, whereas a different subset of floating-point units may be configured to perform 64-bit floating-point operations.

[0306] In at least one embodiment, multiple instances of the GPGPU (1830) may be configured to operate as a computing cluster. In at least one embodiment, the communication used by the computing clusters (1836A-1836H) for synchronization and data exchange varies across embodiments. In at least one embodiment, multiple instances of the GPGPU (1830) communicate through a host interface (1832). In at least one embodiment, the GPGPU (1830) includes an I / O hub (1839) that combines the GPGPU (1830) with a GPU link (1840) that enables direct connection to other instances of the GPGPU (1830). In at least one embodiment, the GPU link (1840) is coupled to a dedicated GPU-to-GPU bridge that enables communication and synchronization between multiple instances of the GPGPU (1830). In at least one embodiment, the GPU link (1840) is coupled to a high-speed interconnect to transmit and receive data to and from other GPGPUs or parallel processors. In at least one embodiment, multiple instances of the GPGPU (1830) are located in separate data processing systems and communicate through network devices accessible via a host interface (1832). In at least one embodiment, the GPU link (1840) may be configured to enable a connection to a host processor in addition to or as an alternative to the host interface (1832).

[0307] In at least one embodiment, the GPGPU (1830) may be configured to train neural networks. In at least one embodiment, the GPGPU (1830) may be used within an inference platform. In at least one embodiment where the GPGPU (1830) is used for inference, the GPGPU (1830) may include fewer computing clusters (1836A-1836H) than when the GPGPU (1830) is used to train neural networks. In at least one embodiment, memory technologies associated with memory (1844A-1844B) may differ between the inference configuration and the training configuration, and higher bandwidth memory technologies are dedicated to the training configurations. In at least one embodiment, the inference configuration of the GPGPU (1830) may support inferring specific instructions. For example, in at least one embodiment, the inference configuration may provide support for one or more 8-bit integer dot product instructions that can be used during inference operations for deployed neural networks.

[0308] Logic (715) is used to perform inference and / or training operations associated with one or more embodiments. Details regarding the logic (715) are provided herein together with FIG. 7a and / or FIG. 7b. In at least one embodiment, the logic (715) may be used in a GPGPU (1830) to infer or predict operations based, at least partially, on weight parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0309] In at least one embodiment, at least one component illustrated or described in connection with FIG. 18a and FIG. 18b is used to implement the techniques and / or functions described in connection with FIG. 1 through FIG. 6. In at least one embodiment, a GPGPU (1830) performs one or more operations of image classification using input data comprising a voxel representation of an environment, at least partially based on a data structure for indicating the voxels to be generated during the conversion to the corresponding voxel representation of the point cloud, as described together with FIG. 1 and as otherwise described in this specification.

[0310] FIG. 19 is a block diagram illustrating a computing system (1900) according to at least one embodiment. In at least one embodiment, the computing system (1900) includes one or more processor(s) (1902) and a processing subsystem (1901) having a system memory (1904) that communicates through an interconnection path that may include a memory hub (1905). In at least one embodiment, the memory hub (1905) may be a separate component within a chipset component or may be integrated within one or more processor(s) (1902). In at least one embodiment, the memory hub (1905) is coupled to an I / O subsystem (1911) through a communication link (1906). In at least one embodiment, the I / O subsystem (1911) includes an I / O hub (1907) that enables the computing system (1900) to receive input from one or more input device(s) (1908). In at least one embodiment, the I / O hub (1907) may enable a display controller, which may be included in one or more processor(s) (1902), to provide outputs to one or more display device(s) (1910A). In at least one embodiment, one or more display device(s) (1910A) coupled with the I / O hub (1907) may include a local, internal, or built-in display device.

[0311] In at least one embodiment, the processing subsystem (1901) includes one or more parallel processor(s) (1912) coupled to the memory hub (1905) via a bus or other communication link (1913). In at least one embodiment, the communication link (1913) may use any number of standard-based communication link technologies or protocols, such as PCI Express, but is not limited thereto, or may be a vendor-specific communication interface or communication fabric. In at least one embodiment, the one or more parallel processor(s) (1912) form a computationally intensive parallel or vector processing system that may include a large number of processing cores and / or processing clusters, such as a many-integrated core (MIC) processor. In at least one embodiment, some or all of the parallel processor(s) (1912) form a graphics processing subsystem capable of outputting pixels to one or more display device(s) (1910A) coupled via the I / O hub (1907). In at least one embodiment, the parallel processor(s) (1912) may also include a display controller and a display interface (not shown) that enable direct connection to one or more display device(s) (1910B). In at least one embodiment, the parallel processor(s) (1912) includes one or more cores, such as the graphics cores (1800) discussed herein.

[0312] In at least one embodiment, a system storage unit (1914) may be connected to an I / O hub (1907) to provide a storage mechanism for a computing system (1900). In at least one embodiment, an I / O switch (1916) may be used to provide an interface mechanism that enables connections between the I / O hub (1907) and other components, such as a network adapter (1918) and / or a wireless network adapter (1919) that may be integrated into the platform, and various other devices that may be added via one or more add-in device(s) (1920). In at least one embodiment, the network adapter (1918) may be an Ethernet adapter or another wired network adapter. In at least one embodiment, the wireless network adapter (1919) may include one or more of Wi-Fi, Bluetooth, NFC (near field communication), or other network devices including one or more wireless radios.

[0313] In at least one embodiment, the computing system (1900) may include other components not explicitly shown, such as USB or other port connections, optical storage drives, video capture devices, etc., and may also be connected to an I / O hub (1907). In at least one embodiment, communication paths interconnecting the various components in FIG. 19 may be implemented using any suitable protocols such as PCI (Peripheral Component Interconnect) based protocols (e.g., PCI-Express), or other bus or point-to-point communication interfaces and / or protocol(s), such as NV-Link high-speed interconnect or interconnect protocols.

[0314] In at least one embodiment, the parallel processor(s) (1912) incorporate a circuit optimized for graphics and video processing, for example, including a video output circuit, and constitute a graphics processing unit (GPU), for example, the parallel processor(s) (1912) includes a graphics core (1800). In at least one embodiment, the parallel processor(s) (1912) incorporate a circuit optimized for general-purpose processing. In at least one embodiment, the components of the computing system (1900) may be integrated with one or more other system elements on a single integrated circuit. For example, in at least one embodiment, the parallel processor(s) (1912), memory hub (1905), processor(s) (1902), and I / O hub (1907) may be integrated into a system-on-chip (SoC) integrated circuit. In at least one embodiment, the components of the computing system (1900) may be integrated within a single package to form a system-in-package (SIP) configuration. In at least one embodiment, at least some of the components of the computing system (1900) may be integrated into a multi-chip module (MCM), which may be interconnected with other multi-chip modules to form a modular computing system.

[0315] Logic (715) is used to perform inference and / or training operations associated with one or more embodiments. Details regarding the logic (715) are provided herein together with FIG. 7a and / or FIG. 7b. In at least one embodiment, the logic (715) may be used in a computing system (1900) to infer or predict operations based, at least partially, on weight parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0316] In at least one embodiment, at least one component illustrated or described with respect to FIG. 19 is used to implement the techniques and / or functions described in relation to FIGS. 1 through 6. In at least one embodiment, parallel processors (1912) perform one or more operations of image classification using input data comprising a voxel representation of an environment based, at least partially, on a data structure to indicate the voxels to be generated during the conversion to the corresponding voxel representation of the point cloud, as described together with FIG. 1 and as otherwise described in this specification.

[0317] processors

[0318] FIG. 20a illustrates a parallel processor (2000) according to at least one embodiment. In at least one embodiment, various components of the parallel processor (2000) may be implemented using one or more integrated circuit devices, such as programmable processors, application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). In at least one embodiment, the illustrated parallel processor (2000) is a variation of one or more parallel processor(s) (1912) illustrated in FIG. 19 according to an exemplary embodiment. In at least one embodiment, the parallel processor (2000) includes one or more graphics cores (1800).

[0319] In at least one embodiment, the parallel processor (2000) includes a parallel processing unit (2002). In at least one embodiment, the parallel processing unit (2002) includes an I / O unit (2004) that enables communication with other devices, including other instances of the parallel processing unit (2002). In at least one embodiment, the I / O unit (2004) may be directly connected to other devices. In at least one embodiment, the I / O unit (2004) connects to other devices through the use of a hub or switch interface, such as a memory hub (2005). In at least one embodiment, the connections between the memory hub (2005) and the I / O unit (2004) form a communication link (2013). In at least one embodiment, the I / O unit (2004) is connected to a host interface (2006) and a memory crossbar (2016), wherein the host interface (2006) receives commands directed to perform processing operations and the memory crossbar (2016) receives commands directed to perform memory operations.

[0320] In at least one embodiment, when the host interface (2006) receives a command buffer through the I / O unit (2004), the host interface (2006) may direct work operations to the front end (2008) to execute these commands. In at least one embodiment, the front end (2008) is coupled with a scheduler (2010) (which may be referred to as a sequencer) configured to distribute commands or other work items to a processing cluster array (2012). In at least one embodiment, the scheduler (2010) ensures that the processing cluster array (2012) is properly configured and in an active state before tasks are distributed to clusters of the processing cluster array (2012). In at least one embodiment, the scheduler (2010) is implemented through firmware logic running on a microcontroller. In at least one embodiment, the microcontroller-implemented scheduler (2010) can be configured to perform complex scheduling and task distribution operations in a rough and fine granular manner, thereby enabling rapid preemption and context switching of threads running in the processing array (2012). In at least one embodiment, the host software can identify workloads to be scheduled on the processing cluster array (2012) through one of a plurality of graphic processing paths. In at least one embodiment, the workloads can then be automatically distributed across the processing array cluster (2012) by the scheduler (2010) logic within the microcontroller including the scheduler (2010).

[0321] In at least one embodiment, the processing cluster array (2012) may include up to "N" processing clusters (e.g., cluster (2014A), cluster (2014B), to cluster (2014N)), where "N" represents a positive integer (which may be an integer "N" different from that used in other drawings). In at least one embodiment, each cluster (2014A-2014N) of the processing cluster array (2012) may execute a large number of concurrent threads. In at least one embodiment, the scheduler (2010) may assign tasks to the clusters (2014A-2014N) of the processing cluster array (2012) using various scheduling and / or task distribution algorithms that may vary depending on the workload occurring for each type of program or computing. In at least one embodiment, scheduling may be dynamically handled by a scheduler (2010) or partially assisted by compiler logic during the compilation of program logic configured for execution by a processing cluster array (2012). In at least one embodiment, different clusters (2014A-2014N) of the processing cluster array (2012) may be allocated for the processing of different types of programs or the execution of different types of computing.

[0322] In at least one embodiment, the processing cluster array (2012) may be configured to perform various types of parallel processing operations. In at least one embodiment, the processing cluster array (2012) is configured to perform general-purpose parallel computing operations. For example, in at least one embodiment, the processing cluster array (2012) may include logic for executing processing tasks including filtering video and / or audio data, performing modeling operations including physics operations, and performing data transformations.

[0323] In at least one embodiment, the processing cluster array (2012) is configured to perform parallel graphics processing operations. In at least one embodiment, the processing cluster array (2012) may include additional logic that supports the execution of these graphics processing operations, including, but is not limited to, texture sampling logic that performs texture operations, as well as tessellation logic and other vertex processing logic. In at least one embodiment, the processing cluster array (2012) may be configured to execute graphics processing-related shader programs, such as vertex shaders, tessellation shaders, geometry shaders, and pixel shaders, but is not limited to. In at least one embodiment, the parallel processing unit (2002) may transfer data from system memory via an I / O unit (2004) for processing. In at least one embodiment, during processing, the transferred data may be stored in on-chip memory (e.g., parallel processor memory (2022)) during processing and then written back to system memory.

[0324] In at least one embodiment, when a parallel processing unit (2002) is used to perform graphics processing, a scheduler (2010) may be configured to divide the processing workload into tasks of approximately equal size to better enable the distribution of graphics processing operations to a plurality of clusters (2014A-2014N) of a processing cluster array (2012). In at least one embodiment, parts of the processing cluster array (2012) may be configured to perform different types of processing. For example, in at least one embodiment, a first part may be configured to perform vertex shading and topology generation, a second part may be configured to perform tessellation and geometry shading, and a third part may be configured to perform pixel shading or other screen space operations to produce a rendered image for display. In at least one embodiment, intermediate data produced by one or more of the clusters (2014A-2014N) may be stored in buffers to allow the intermediate data to be transmitted between the clusters (2014A-2014N) for further processing.

[0325] In at least one embodiment, the processing cluster array (2012) may receive processing tasks to be executed via a scheduler (2010) that receives commands defining processing tasks from a front end (2008). In at least one embodiment, the processing tasks may include index values ​​of data to be processed, e.g., surface (patch) data, primitive data, vertex data, and / or pixel data, as well as state parameters and commands defining how the data should be processed (e.g., which program should be executed). In at least one embodiment, the scheduler (2010) may be configured to retrieve index values ​​corresponding to the tasks or receive index values ​​from the front end (2008). In at least one embodiment, the front end (2008) may be configured to ensure that the processing cluster array (2012) is configured to be in an active state before a workload specified by incoming command buffers (e.g., batch buffers, push buffers, etc.) is initiated.

[0326] In at least one embodiment, each of one or more instances of a parallel processing unit (2002) may be coupled with a parallel processor memory (2022). In at least one embodiment, the parallel processor memory (2022) may be accessed through a memory crossbar (2016) capable of receiving memory requests from the processing cluster array (2012) as well as from an I / O unit (2004). In at least one embodiment, the memory crossbar (2016) may access the parallel processor memory (2022) through a memory interface (2018). In at least one embodiment, the memory interface (2018) may include a plurality of partition units (e.g., partition unit (2020A), partition unit (2020B), to partition unit (2020N)) which may each be coupled to a portion (e.g., a memory unit) of the parallel processor memory (2022). In at least one embodiment, the number of partition units (2020A-2020N) is configured to be equal to the number of memory units, such that the first partition unit (2020A) has a corresponding first memory unit (2024A), the second partition unit (2020B) has a corresponding memory unit (2024B), and the Nth partition unit (2020N) has a corresponding Nth memory unit (2024N). In at least one embodiment, the number of partition units (2020A-2020N) may not be equal to the number of memory units.

[0327] In at least one embodiment, the memory units (2024A-2024N) may include various types of memory devices, such as dynamic random access memory (DRAM) or graphics random access memory, such as synchronous graphics random access memory (SGRAM), including graphics double data rate (GDDR) memory. In at least one embodiment, the memory units (2024A-2024N) may also include 3D stacked memory, including but not limited to high bandwidth memory (HBM), HBM2e, or HDM3. In at least one embodiment, render targets, such as frame buffers or texture maps, may be stored across the memory units (2024A-2024N) so that partition units (2020A-2020N) allow writing in parallel to parts of each render target to efficiently use the available bandwidth of the parallel processor memory (2022). In at least one embodiment, a local instance of parallel processor memory (2022) may be excluded for an integrated memory design that utilizes system memory together with local cache memory.

[0328] In at least one embodiment, any one of the clusters (2014A-2014N) of the processing cluster array (2012) can process data to be written to any of the memory units (2024A-2024N) within the parallel processor memory (2022). In at least one embodiment, the memory crossbar (2016) may be configured to transmit the output of each cluster (2014A-2014N) to any partition unit (2020A-2020N) or to another cluster (2014A-2014N) capable of performing additional processing operations regarding the output. In at least one embodiment, each cluster (2014A-2014N) may communicate with the memory interface (2018) through the memory crossbar (2016) to read from or write to various external memory devices. In at least one embodiment, the memory crossbar (2016) has a connection to a memory interface (2018) for communicating with an I / O unit (2004), as well as a connection to a local instance of parallel processor memory (2022), so that processing units within different processing clusters (2014A-2014N) can communicate with system memory or other memory that is not local to the parallel processing unit (2002). In at least one embodiment, the memory crossbar (2016) may use virtual channels to separate traffic streams between the clusters (2014A-2014N) and the partition units (2020A-2020N).

[0329] In at least one embodiment, multiple instances of the parallel processing unit (2002) may be provided on a single add-in card, or multiple add-in cards may be interconnected. In at least one embodiment, different instances of the parallel processing unit (2002) may be configured to interact with each other even if the different instances have different numbers of processing cores, different amounts of local parallel processor memory, and / or other configuration differences. For example, in at least one embodiment, some instances of the parallel processing unit (2002) may include higher precision floating-point units than other instances. In at least one embodiment, systems incorporating one or more instances of the parallel processing unit (2002) or the parallel processor (2000) may be implemented in various configurations and form factors, including but not limited to desktops, laptops or handheld personal computers, servers, workstations, game consoles, and / or embedded systems.

[0330] FIG. 20b is a block diagram of a partition unit (2020) according to at least one embodiment. In at least one embodiment, the partition unit (2020) is an instance of one of the partition units (2020A-2020N) of FIG. 20a. In at least one embodiment, the partition unit (2020) includes an L2 cache (2021), a frame buffer interface (2025), and an ROP (2026) (raster operation unit). In at least one embodiment, the L2 cache (2021) is a read / write cache configured to perform loading and saving operations received from the memory crossbar (2016) and the ROP (2026). In at least one embodiment, read misses and urgent write-back requests are output to the frame buffer interface (2025) by the L2 cache (2021) for processing. In at least one embodiment, updates may also be transmitted to the frame buffer through the frame buffer interface (2025) for processing. In at least one embodiment, the frame buffer interface (2025) interfaces with one of the memory units in the parallel processor memory, such as the memory units (2024A-2024N) of FIG. 20a (e.g., in the parallel processor memory (2022)).

[0331] In at least one embodiment, the ROP (2026) is a processing unit that performs raster operations such as stenciling, z-testing, blending, etc. In at least one embodiment, the ROP (2026) then outputs processed graphic data that is stored in graphic memory. In at least one embodiment, the ROP (2026) includes compression logic that compresses depth or color data written to memory and decompresses depth or color data read from memory. In at least one embodiment, the compression logic may be lossless compression logic using one or more of a plurality of compression algorithms. In at least one embodiment, the type of compression performed by the ROP (2026) may vary based on the statistical characteristics of the data to be compressed. For example, in at least one embodiment, delta color compression is performed on depth and color data on a per-tile basis.

[0332] In at least one embodiment, the ROP (2026) is contained within each processing cluster (e.g., clusters (2014A-2014N) of FIG. 20a) instead of within the partition unit (2020). In at least one embodiment, read and write requests for pixel data are transmitted through the memory crossbar (2016) instead of the pixel fragment data. In at least one embodiment, the processed graphic data may be displayed on a display device such as one of the one or more display device(s) (1910) of FIG. 19, routed for further processing by the processor(s) (1902), or routed for further processing by one of the processing entities within the parallel processor (2000) of FIG. 20a.

[0333] FIG. 20c is a block diagram of a processing cluster (2014) within a parallel processing unit according to at least one embodiment. In at least one embodiment, the processing cluster is an instance of one of the processing clusters (2014A-2014N) of FIG. 20a. In at least one embodiment, the processing cluster (2014) may be configured to execute many threads in parallel, where "thread" refers to an instance of a specific program executed on a specific set of input data. In at least one embodiment, SIMD (single-instruction, multiple-data) instruction issuance techniques are used to support the parallel execution of a large number of threads without providing multiple independent instruction units. In at least one embodiment, SIMT (single-instruction, multiple-thread) techniques are used to support the parallel execution of a large number of generally synchronized threads by using a common instruction unit configured to issue instructions to a set of processing engines within each of the processing clusters.

[0334] In at least one embodiment, the operation of the processing cluster (2014) may be controlled through a pipeline manager (2032) that distributes processing tasks to SIMT parallel processors. In at least one embodiment, the pipeline manager (2032) receives instructions from the scheduler (2010) of FIG. 20a and manages the execution of these instructions through a graphics multiprocessor (2034) and / or a texture unit (2036). In at least one embodiment, the graphics multiprocessor (2034) is an exemplary instance of a SIMT parallel processor. However, in at least one embodiment, various types of SIMT parallel processors of different architectures may be included in the processing cluster (2014). In at least one embodiment, one or more instances of the graphics multiprocessor (2034) may be included in the processing cluster (2014). In at least one embodiment, a graphics multiprocessor (2034) can process data, and a data crossbar (2040) can be used to distribute the processed data to one of a plurality of possible destinations, including other shader units. In at least one embodiment, a pipeline manager (2032) can facilitate the distribution of the processed data by specifying the destination for the processed data to be distributed through the data crossbar (2040).

[0335] In at least one embodiment, each graphics multiprocessor (2034) within the processing cluster (2014) may include the same set of function execution logic (e.g., arithmetic logic units, load-store units, etc.). In at least one embodiment, the function execution logic may be configured in a pipelined manner so that new instructions can be issued before previous instructions are completed. In at least one embodiment, the function execution logic supports various operations including integer and floating-point arithmetic, comparison operations, Boolean operations, bit shifting, and computing of various algebraic functions. In at least one embodiment, the same function unit hardware may be utilized to perform different operations, and any combination of function units may exist.

[0336] In at least one embodiment, instructions transmitted to the processing cluster (2014) constitute threads. In at least one embodiment, a set of threads executed across a set of parallel processing engines is a thread group. In at least one embodiment, the thread group executes a common program on different input data. In at least one embodiment, each thread within the thread group may be assigned to a different processing engine within the graphics multiprocessor (2034). In at least one embodiment, the thread group may include fewer threads than the number of processing engines within the graphics multiprocessor (2034). In at least one embodiment, when the thread group includes fewer threads than the number of processing engines, one or more of the processing engines may be idle during the cycles in which the thread group is being processed. In at least one embodiment, the thread group may also include more threads than the number of processing engines within the graphics multiprocessor (2034). In at least one embodiment, when the thread group includes more threads than the number of processing engines within the graphics multiprocessor (2034), processing may be performed over consecutive clock cycles. In at least one embodiment, multiple thread groups can be executed simultaneously on a graphics multiprocessor (2034).

[0337] In at least one embodiment, the graphics multiprocessor (2034) includes an internal cache memory for performing loading and saving operations. In at least one embodiment, the graphics multiprocessor (2034) may use cache memory (e.g., L1 cache (2048)) within a processing cluster (2014) without using the internal cache. In at least one embodiment, each graphics multiprocessor (2034) also has access to L2 caches within partition units (e.g., partition units (2020A-2020N) of FIG. 20a) that are shared am...

Claims

Claim 1 A processor comprising one or more circuits that, as a processor, generate one or more voxels using one or more point clouds based at least partially on one or more point locations and represent said one or more voxels in one or more data structures. Claim 2 A processor according to claim 1, wherein the one or more circuits generate the one or more data structures such that the one or more hash tables store the one or more point locations as keys and the one or more representations of the one or more voxels as one or more corresponding values. Claim 3 In claim 1, the processor, wherein the one or more circuits generate the one or more data structures using an amount of memory at least partially based on the number of point locations in the one or more point clouds. Claim 4 A processor according to claim 1, wherein the one or more circuits access one or more data values ​​using the one or more data structures and generate one or more feature values ​​of the one or more voxels based at least partially on the one or more data values. Claim 5 A processor according to claim 1, wherein the one or more circuits store one or more data values ​​representing features associated with the one or more point locations within one or more memory locations to be accessed based at least partially on data stored in the one or more data structures. Claim 6 In claim 1, the one or more circuits are a processor that traverses over the one or more point locations to generate one or more feature values ​​of the one or more voxels using the one or more data structures. Claim 7 In claim 1, the one or more circuits input one or more indications of the one or more point locations into a hash function and output one or more indications of the one or more voxels, a processor. Claim 8 A system comprising one or more processors that generate one or more voxels using one or more point clouds based at least partially on one or more point locations and display said one or more voxels in one or more data structures. Claim 9 In claim 8, the system generates the one or more data structures such that the one or more processors include at least one hash table based at least partially on one or more arrays of one or more point locations and one or more arrays of one or more representations of one or more voxels. Claim 10 In claim 8, the system wherein the one or more processors generate the one or more data structures using an amount of memory that is at least twice the number of the one or more point locations. Claim 11 In claim 8, the system comprises one or more processors generating one or more data structures such that the one or more tables for indexing the one or more voxels as a function of the one or more point locations. Claim 12 In claim 8, the system wherein the one or more processors represent the one or more voxels based at least partially on one or more representations of memory locations used to store one or more representations of the one or more voxels. Claim 13 In claim 8, the system wherein the one or more processors traverse over the one or more point locations to generate one or more average feature values ​​of the one or more voxels using the one or more data structures. Claim 14 A system according to claim 8, wherein the one or more processors input one or more marks of one or more point locations into a hash function to output one or more marks of one or more memory locations used to store one or more marks of one or more of the one or more of the one or more of the voxels. Claim 15 A method comprising the step of generating one or more voxels using one or more point clouds based at least partially on one or more point locations and displaying one or more voxels in one or more data structures. Claim 16 A method according to claim 15, further comprising the step of generating one or more data structures to include one or more hash tables using one or more point locations as keys for indicating one or more voxels to be generated. Claim 17 A method according to claim 15, further comprising the step of creating one or more data structures to use an amount of memory at least partially based on the number of points in the one or more point clouds. Claim 18 A method according to claim 15, further comprising the step of accessing one or more representations of memory locations used to store one or more representations of one or more of the above-mentioned voxels. Claim 19 A method according to claim 15, further comprising the step of traversing over one or more point locations to generate one or more sum feature values ​​of one or more voxels using one or more data structures. Claim 20 A method according to claim 15, further comprising the step of accessing one or more data values ​​using one or more data structures and generating one or more voxels based at least partially on one or more data values.

Images