Real-time multi-view map generation using neural networks

Abstract

Several examples disclose systems and methods related to real-time, multi-view map generation using neural networks. A system can receive sensor images of an environment, such as images from one or more camera, radar, lidar, and / or ultrasonic sensors. The system can process the sensor images using one or more neural networks, such as neural networks implementing attention structures, to detect features in the environment, such as lane lines, lane dividers, stop lines, or boundaries. The system can display the features in various views, including top-down / bird's-eye views. The system can provide these displays for operations, including map generation, map updates, perception, and object detection.

Description

BACKGROUNDMachine learning models, such as neural networks, may be used to process image data from cameras and other sensors to detect objects or other features in environments. This may be performed, for example, to determine a map of an environment. However, such approaches may lack realism or quality, including when used for real-time map generation.SUMMARYEmbodiments of the present disclosure relate to systems and methods for generating three-dimensional (3D) map data using neural networks, including neural networks having transformer architectures. For example, the systems may process multiple images to include depth information into the image processing pipeline and may use different attention modules to allow 3D information regarding features to be efficiently captured and mapped to different frames of reference, such as a top-down / bird's-eye view frame of reference. Compared to conventional systems, such as those described above, systems and methods according to the present disclosure may enable more realistic, qualitative generation of maps or other representations of features and may enable such aspects with greater computational efficiency.At least one aspect relates to a processor. The processor may include one or more circuits for receiving a first sensor image captured at a first time and a second sensor image captured at a second time. The one or more circuits may determine, using a neural network and based at least on the first sensor image and the second sensor image, one or more features represented by the first sensor image and the second sensor image. The one or more circuits may determine, using the neural network, a grid of the scene in which the one or more features are each associated with a cell of the grid. The one or more circuits may include at least one of (i) associating the grid with a map data structure or (ii) presenting the grid using a display device.In some implementations, the one or more circuits may determine the one or more features using at least one of radio detection and ranging (RADAR) data, light detection and ranging (LIDAR) data, or ultrasonic data corresponding to at least one of the first sensor image or the second sensor image. In some implementations, the grid includes a two-dimensional representation of the scene in a top-down frame of reference. In some implementations, the one or more circuits may determine, for each feature of the one or more features, a polyline representing the feature. The polyline may include a plurality of points indicating a plurality of line segments. The one or more circuits may associate the feature with the cell by associating the polyline with the cell.In some implementations, the one or more circuits may provide, as input to the neural network, a position representation of at least one of a camera center of the first sensor image, a camera center of the second sensor image, a vector of a ray to a feature of the first sensor image, or a vector of a ray to a feature of the second sensor image. In some implementations, the one or more circuits may include a feature generator to convert image data of the first sensor image and the second sensor image into a plurality of tokens in a latent data space. In some implementations, the one or more circuits may include an encoder cross-attention processor to process the plurality of tokens and a latent data representation maintained by one or more self-attention modules. In some implementations, the one or more circuits may include a decoder cross-attention processor to process an intermediate output of the neural network and the latent data representation to determine the grid of the scene.In some implementations, the one or more circuits may associate a height of the feature and / or a class of the feature with the cell. In some implementations, the first sensor image and the second sensor image may include camera data.At least one aspect relates to a processor. The processor may include one or more circuits for receiving training data including a first sensor image captured at a first time and a second sensor image captured at a second time, wherein at least one feature is associated with the first sensor image and the second image. The one or more circuits may determine, using at least one neural network and based at least on the first sensor image and the second sensor image, an estimated output indicative of at least one position of at least one estimated feature. The one or more circuits may update the at least one neural network based at least on the estimated output and the at least one feature.In some implementations, the first sensor image and the second sensor image include at least one of camera data, RADAR data, LIDAR data, or ultrasound data. In some implementations, the at least one feature includes at least one of a lane line, a lane divider, a waiting line, or a boundary structure. In some implementations, the at least one neural network includes an encoder attention network, a plurality of latent space attention networks, and a decoder attention network.At least one aspect relates to a method. The method may include receiving, using one or more processors, a first sensor image captured at a first time and a second sensor image captured at a second time. The method may include determining, using the one or more processors and a neural network and based at least on the first sensor image and the second sensor image, one or more features represented by the first sensor image and the second sensor image. The method may include determining, using the one or more processors and the neural network, a grid of the scene in which the one or more features are each associated with a cell of the grid. The method may include at least one of (i) mapping, using the one or more processors, the grid to a map data structure, or (ii) presenting, using the one or more processors and a display device, the grid.In some implementations, the method includes determining, using the one or more processors, the one or more features using at least one of RADAR data, LIDAR data, or ultrasonic data corresponding to at least one of the first sensor image or the second sensor image. In some implementations, the method includes providing, using the one or more processors, as input to the neural network, a position representation of at least one of a camera center of the first sensor image, a camera center of the second sensor image, a vector of a ray to a feature of the first sensor image, or a vector of a ray to a feature of the second sensor image.In some implementations, the neural network includes a feature generator to convert image data of the first sensor image and the second sensor image into a plurality of tokens in a latent data space, an encoder cross-attention processor to process the plurality of tokens and a latent data representation maintained by one or more self-attention modules, and a decoder cross-attention processor to process an intermediate output of the neural network and the latent data representation to determine the grid of the scene.In some implementations, the grid includes a two-dimensional representation of the scene in a top-down frame of reference. The method may include determining, using the one or more processors, for each feature of the one or more features, a polyline representing the feature. The polyline may include a plurality of points indicating a plurality of line segments, wherein the one or more circuits are to associate the feature of the cell with the cell by associating the polyline with the cell.In some implementations, the method includes associating, using the one or more processors, a height of the feature and / or a class of the feature with the cell. In some implementations, the first sensor image and the second sensor image include camera data.The processors, systems, and / or methods described herein may be implemented by a system associated with an autonomous or semi-autonomous machine (e.g., an in-vehicle infotainment system) and / or a system for performing simulation operations and / or a system for performing digital twin operations and / or a system for performing light transport simulation and / or a system for performing collaborative content creation for 3D assets and / or a system for performing deep learning operations and / or a system implemented using an edge device and / or a system implemented using a robot, and / or a system for generating or presenting virtual reality (VR) content, augmented reality (AR) content and / or mixed reality (MR) content and / or a system for performing conversation AI operations and / or a system for generating synthetic data and / or a system including one or more virtual machines (VMs), and / or a system implemented at least partially in a data center and / or a system implemented at least partially using cloud computing resources.BRIEF DESCRIPTION OF THE DRAWINGSThe present systems and methods for real-time multi-view map generation using neural networks are described in detail below with reference to the accompanying drawing figures, wherein: FIG. 1 is a block diagram of an example system for real-time multi-view (multi-view) map generation using neural networks, in accordance with some embodiments of the present disclosure; FIG. 2 is a schematic diagram of an example 3D map representation of an environment that may be generated by the example system of FIG. 1, in accordance with some embodiments of the present disclosure; FIG. 3 is a flow diagram of an example of a method for real-time multi-view map generation, in accordance with some embodiments of the present disclosure; FIG. 4 is a block diagram of an example content streaming system suitable for use in implementing some embodiments of the present disclosure; FIG. 5 is a block diagram of an example computing device suitable for use in implementing some embodiments of the present disclosure; and FIG. 6 is a block diagram of an example data center suitable for use in implementing some embodiments of the present disclosure.DETAILED DESCRIPTIONSystems and methods are disclosed that relate to systems and methods for generating three-dimensional (3D) map data using neural networks, including neural networks having transformer architectures, for example. The systems and methods may be implemented to enable temporal, real-time, multi-view (multi-view) map generation, as well as to determine high resolution maps for downstream use, such as perception tasks, or to perform dynamic object detection. The systems and methods may be implemented to enable more realistic and qualitative feature detection and representation, including by capturing depth information to avoid relying on flat world assumption regarding the shapes, extents, and / or positions of 3D objects in image-based representations.Some systems may process image data from cameras and other sensors, such as RADAR or LIDAR, to detect features in an environment. For example, image data may be processed to detect features useful for vehicles, such as lane dividers, road boundaries, waiting lines, traffic signs, etc. Such approaches may rely on detecting features in the camera frame data (e.g., two-dimensional (2D) image data) and then projecting the features into 3D using various assumptions about relationships between 2D and 3D representations of the features. However, such approaches may lack quality, realism, and / or the ability to be performed in real-time, such as for real-time image processing for use by a vehicle sensor system / autonomous vehicle controller. Moreover, such approaches may require fine-tuned post-processing, which may increase latency or otherwise make calculations more complex.Systems and methods according to the present disclosure may use neural networks configured to capture features of environments from camera and other sensor data. The neural networks may include an encoder cross-attention module / processor to relate input data to latent data (e.g., information held in memory by the network), such as to allow the neural network to select tokens of interest (tokens may represent portions of image data, such as after feature generation). The neural networks may include one or more self-attention modules for detecting relationships between tokens. The neural networks may include a decoder cross-attention module / processor to query the neural networks for outputs to enable the neural networks to learn feature locations to generate a final output data structure.For example, the system may process multiple images at different times to acquire depth information; the system may also process other sensor data, such as RADAR / LIDAR / ultrasound data, to acquire depth and other information regarding features. Image / sensor data may be feature generated to reduce noise and processing requirements (e.g., by reducing the number of tokens of image data for further processing).Inductive priors, such as camera positions and beam vectors, may be assigned or added to the input image frames (e.g., after being feature generated by a convolutional neural network), which may allow the model to triangulate features across images. The system may learn the locations of features (e.g., rather than determining locations in pixel space or a latent feature space) by allowing the system to process image data without assigning position / coordinate information to tokens of image data; intermediate outputs may be queried to facilitate the position detection process. For example, the system may learn position encodings so that the system may associate / correlate tokens from different sensor streams or image frames. Various inputs and / or inductive priors, such as position data, may be encoded using FFT (sin / cos) encoding.The system may determine outputs that include representations of the features in the environment. For example, lane lines and other features may be represented as polylines, which may include one or more vertices (e.g., Bezier curve points / control points) that may represent one or more line segments to represent a shape of the features. The representations may be determined in different frames of reference, such as a top-down / bird's eye. For example, representations may be determined in a grid data structure in which polylines are associated with one or more cells (e.g., anchors) of the grid, such as a cell closest to a center of the polyline (associating the polyline with a single cell may reduce processing requirements and also allow the system to be aware of the portions of the polylines that may otherwise be occluded, and further allow the data representation in the cell to reduce a full representation of the feature in the entire scene). The system may encode confidence as to whether a polyline is associated with the cell (during training, the confidence may be set to 100 percent because the data represents ground truth). The system may encode a class of the feature in the cell. The system may also output depth information, such as by identifying or outputting derived inverse disparity that may be converted to depth information.The systems and methods described herein may be used for a variety of purposes, including, but not limited to, synthetic data and map generation, machine control, machine travel, machine driving, model training, perception, augmented reality, virtual reality, mixed reality, robotics, safety and monitoring, simulation and digital twinning, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation, and / or digital twinning, data center processing, conversation AI, light transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation for 3D assets, Cloud computing and / or any other suitable applications.Disclosed embodiments may be included in a variety of different systems, such as systems for performing synthetic data generation operations, automotive systems (e.g., an autonomous or semi-autonomous machine control system, an autonomous or semi-autonomous machine perception system), systems implemented using a robot, air systems, medical systems, boat systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using an edge device, systems including one or more virtual machines (VMs), systems at least partially implemented in a data center, Systems for performing conversation AI operations, systems for performing light transport simulation, systems for performing collaborative content creation for 3D assets, systems implemented at least in part using cloud computing resources, and / or other types of systems.FIG. 1 is an example computing environment including a system 100, in accordance with some embodiments of the present disclosure. It should be understood that these and other arrangements described herein are presented by way of example only. Other arrangements and elements (e.g., machines, interfaces, functions, orders, groupings of functions, etc.) may be used in addition to or in place of those shown, and some elements may be omitted altogether. Further, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in conjunction with other components and in any suitable combination and position. Various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in a memory. The system 100 may include any function, model (e.g., machine learning model), operation, routine, logic, or instructions for performing functions, such as configuring machine learning models 104 as described herein, such as configuring machine learning models to operate as transformer-based models, attention-based models, or various combinations thereof, including for 3D map generation.In some implementations, the system 100 performs operations in a 3D reference frame. The 3D reference frame may be a reference frame of a sensor (e.g., a sensor used to acquire image data 108) or a platform to which the sensor is coupled, such as a rig or vehicle. For example, the 3D reference frame may be an X-Y-Z coordinate reference frame in which (positive) X corresponds to a forward direction, (positive) Y corresponds to a leftward direction, and (positive) Z corresponds to a Z direction.The system 100 may train, update, or configure one or more models 104. The models 104 may include machine learning models or other models that may generate target outputs based on various types of input. The models 104 may include one or more neural networks. The neural network or networks may include an input layer, an output layer, and / or one or more intermediate layers, such as hidden layers, each of which may have respective nodes. The system 100 may train / update the neural network by modifying or updating one or more parameters, such as weights and / or distortions, of various nodes of the neural network in response to evaluating estimated outputs of the neural network.The models 104 may be or include various neural network models, including models operable / implemented to process or generate data including, but not limited to, image data, video data, text data, voice data, audio data, or various combinations thereof. The models 104 may include one or more transformers, detection transformers (DETRs), deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), long short term memory (LSTM) models, other network types, or various combinations thereof. The models 104 may include generative models, such as generative adventrial networks (GANs), Markov decision processes, variational autoencoders (VAEs), Bayesian networks, transformer models, Truer models, autoregressive models, autoregressive encoder models (e.g., a model including an encoder for generating a latent representation (e.g., in an embedding space) of an input to the model (e.g., a representation of a different dimensionality than the input) and / or a decoder for generating an output representing the input from the latent representation), or various combinations thereof. As further described herein, the models 104 may include neural network models including, but not limited to, feature creator 112, coder 124, attention processors 128, 144, decoder 152, and / or attention processor 156.The system 100 may perform operations on image data 108, which may include various forms of image-based sensor data. The image data 108 may include, for example and without limitation, camera data, ultrasonic sensor data, LIDAR data, RADAR data, or various combinations thereof. The image data 108 may be retrieved from any of a variety of databases or sensors (e.g., LIDAR sensors, RADAR sensors, ultrasonic sensors, cameras, image capture devices, etc.). For example, the image data 108 may be retrieved from sensors or may be retrieved from one or more databases in which image data is stored after collection. The image data 108 may be retrieved from sensors coupled to vehicles, such as autonomous vehicles.In some implementations, the image data 108 may include images (e.g., sensor images) of multiple views and / or multiple times. For example, the image data 108 may include a plurality of images of multiple poses, such as multiple sensors coupled to a vehicle at different locations. The poses of the images may correspond to an origin and a direction (e.g., perspective) of the images. This may include, for example and without limitation, front, left, and right poses of sensors on a vehicle. The images of multiple poses may be acquired at the same or different times. The image data 108 may include images captured at multiple times for each of one or more respective sensors and / or poses. For example, the image data 108 may include one or more of a first sensor image of (e.g., captured by) a first pose at a first time, a second sensor image of a second pose at the first time, a third sensor image of a third pose at the first time, a fourth sensor image of the first pose at a second time, a fifth sensor image of the second pose at the second time, a sixth sensor image of the third pose at the second time, etc.The image data 108 may include RADAR and / or LIDAR data from the same or different poses and times as images from cameras. The RADAR or LIDAR data may include top-down perspective view data or bird's eye view (BEV) data. In some implementations, the image data 108 includes camera image frames from a front camera (e.g., 120-degree field of view), a left side, a right side, a rear left and a rear right camera (e.g., 200-degree field of view), such as to include 6 frames at each time step (e.g., at time T, time T+1, etc.); for example, the camera data of the image data 108 may be from the same or different perspectives as the RADAR or LIDAR data of the image data 108.The image data 108 may include information related to the sensor(s) that / s) that captured the images associated with the image data 108. For example, the image data 108 may include information such as a center of the sensors (e.g., camera center) or a direction of the sensors (e.g., camera beam vector) that / have captured the images associated with the images. As further described herein, this may allow the system 100 to configure the models 104 to learn location information regarding features (e.g., rather than relying directly on location information in the images 108).The system 100 may include at least one feature generator 112 (e.g., feature generator processor). The feature generator 112 may include one or more instructions, rules, heuristics, models, policies, functions, algorithms, etc., to perform operations including identifying features (e.g., representations of objects or other structures in the environment) from image data 108. The feature generator 112 may include a neural network such as a CNN. The feature generator 112 may process the image data 108 to generate a modified representation of the image data 108, such as the input device 116. The input array 116 may include a plurality of tokens, each token indicating a respective feature shown in the image data 108.The feature generator 112 may be trained using image data 108 (or images analogous to the image data 108) labeled with indications of features to be captured, which allows the feature generator 112 to determine the input arrangement 116 as an image data structure indicative of features. In some implementations, the feature generator 112 determines that the input array 116 has a different dimensionality (e.g., dimensions with fewer pixels per dimension) than the image data 108. The feature generator 112 may process the image data 108 to determine a plurality of tokens representing features of the image data 108. In some implementations, the feature generator 112 may downsample the image data 108 to determine the input array 116. For example, the image data 108 may have a first size in pixel space (e.g., 960×480° pixels) and the input array 116 may have a second size that is less than the first size (e.g., 480×240 pixels).In some implementations, the system 100 uses at least some different subsets of the image data 108. For example, the system 100 may use a first subset, such as a first stack, of the image data 108 to determine a first output (e.g., output 164) or intermediate outputs from image data 108, and a second subset, such as a second stack, of the image data 108 to determine a second output 164. The first subset and the second subset may originate from the same or different databases of image data 108, such as different databases with different levels of public accessibility. The first subset may be a training dataset and the second subset may be a test or validation subset. The databases may include data of different types of annotations, such as annotations indicating different classes or categories of objects or features.In some implementations, the system 100 modifies the input device 116 using one or more auxiliary data 120. The supplemental data 120 may be used as priors, such as inductive priors, to enable the system 100 to learn various information associated with features represented by the image data 108; for example, the supplemental data 120 may enable the system 100 to learn positions of features (e.g., rather than relying directly on position information represented by pixel positions of features in the image data 108 and / or determine the positions of features in the pixel space or latent feature space; this may enable the system 100 to operate more effectively in the frame of reference of the output 164). The supplemental data 120 may include camera information regarding the image data 108, such as a position of an origin or center of the image capture device that captured the image data 108, a beam vector (e.g., direction vector) that represents a direction or perspective from the image capture device to the position of the image data 108, or various combinations thereof. In some implementations, the supplemental data 120 indicates directions (e.g., camera beam vectors) to respective pixels or groups of pixels of the image data 108, which may be used to position encode portions of the image data 108 that are processed by the feature generator 112 into the input array 116. The supplemental data 120 may include or be retrieved from calibration data relating to the image capture devices, such as to enable the system 100 to relate the image data 108 to be in a common frame of reference. The system 100 may perform various position encodings, such as fast Fourier transform (FFT) and / or sin / cos encodings of the auxiliary data 120. The system 100 may add the auxiliary data 120 to the input device 116 and / or combine the input device 116 with the auxiliary data 120, such as by concatenating the auxiliary data 120 with the input device 116.With continued reference to FIG. 1, the system 100 may include at least one encoder 124, such as an encoder processor. The encoder 124 may include one or more instructions, rules, heuristics, models, neural networks, transformer neural networks, policies, functions, algorithms, etc., to perform operations including vector operations and / or applying attention and / or cross-attention to one or more inputs provided to the encoder 124. For example, the encoder 124 may perform operations including compressing or otherwise modifying the input array 116 into a latent space (e.g., from the pixel space of the image data 108) for processing 136 the latent space.In some implementations, the encoder 124 is or includes at least one attention processor 128. Attention processor 128 may include one or more neural networks, such as transformer neural networks configured to perform operations such as additive attention, dot product attention, scaled dot product attention, or combinations thereof. As shown in FIG. 1, the encoder 124 may receive the input array 116 as a key vector K and may receive the input array 116 as a value vector V and a latent array 132 as a query vector Q to apply it as input to the attention processor 128. As mentioned above, the input array 116 may include an array of elements representing one or more features of the image data 108, and may be arranged in an order that is independent of the positioning of features in the image data 108 (and each element of the input array 116 may be encoded with position data indicated by supplemental data 120). The latent array 132 may be retrieved from the latent array 140 of the latent space processing 136. In some implementations, attention processor 128 may enable system 100 to detect feature-like patches in input array 116, including, without limitation, track-like patches, and associate the feature-like patches (vectors of) with latent array 140. By processing the input array 116 and the latent array 140 using attention, the system 100 may capture attention / similarity between the vector representations of the feature-like patches, which may allow the system 100 to learn how features in multiple images (e.g., from different cameras or times) are similarly presented. By determining the input arrangement 116, in some implementations, the system 100 may allow the attention processor 128 to triangulate positions of similar / identical feature-like patches using supplemental data 120 (e.g., position coding).The machine learning models 104 may include one or more neural networks, such as attention processors 144, to perform latent space processing 136. For example, as shown in FIG. 1, attention processor 128 may determine a latent array 140 as output in response to processing of input array 116 and latent array 132. The latent assembly 140 (and the latent assembly 148) may have a similar or identical structure to the latent assembly 132.As shown in FIG. 1, the latent space processing 136 may include a plurality of attention processors 144, which may be neural networks (e.g., transformer neural networks) having a similar or identical structure to the attention processor 128. The attention processors 144 may be arranged in a sequence (e.g., of multiple attention blocks) such that the output of the latent space (which may have the same structure as the latent array 140) of a first attention processor 144 is provided as inputs (e.g., query Q, key K, and / or value V inputs) to a second attention processor 144 downstream of the first attention processor 144 to determine a latent array 148.The system 100 may determine at least one output polling arrangement 160 based at least on [ ]. The output query array 160 may represent a plurality of output elements, such as output anchors. The output anchors may be arranged in a top down / BEV reference frame. For example, the output query array 160 may be arranged in X-Y coordinates (e.g., real distances of items), where Z coordinate data (e.g., height data) is assigned to respective items. In some implementations, the output interrogation arrangement 160 may have a center offset from a center of the frame of reference of the image data 108, which may allow the processing of the information from the image data 108 to be focused or weighted on information in front of the center of the platform that includes the sensor(s) that captured / have the image data 108.The output anchors may represent features where one or more features intersect a respective anchor. In some implementations, the system 100, using the output query array 160, may represent features using one or more line segments, such as polyline segments. For example, the polyline segment may include at least one of a plurality of vertices or a plurality of line segments connecting respective pairs of vertices of the plurality of vertices.With continued reference to FIG. 1, the system 100 may include at least one decoder 152. The decoder 152 may include one or more instructions, rules, heuristics, models, neural networks, transformer neural networks, policies, functions, algorithms, etc., to perform operations including vector operations and / or applying attention and / or cross-attention to one or more inputs provided to the decoder 152. Decoder 152 may perform operations to determine an output 164, such as a top-down perspective and / or a BEV representation of image data 108. For example, decoder 152 may perform cross-attention between output interrogation assembly 160 and latent assembly 148, which may allow system 100 to identify features from the latent spatial representation of the features of image data 108 to assign features using output interrogation assembly 160. In some implementations, presenting the output query array 160 using output anchors assigned polyline segments may reduce computational requirements for determining, maintaining, and updating the output query array 160 relative to maintaining explicit BEV tensor data structures. The output 164 may be a grid having a plurality of cells at respective locations formed by the coordinates of the grid.The system 100 may determine the output 164 to represent various features or classes of environmental features represented by the image data 108, including, for example and without limitation, segments of lanes, road boundaries, vehicles, lines of vehicles, curbs, barriers, medians, or various other features. In some implementations, the system 100 selects an apex of a plurality of apexes of the line segment of a given feature represented by the output query array 160 and assigns the line segment to the output query array 160 and / or the output 164 to a coordinate (e.g., X-Y coordinate) of the output query array 160 and / or the output 164 corresponding to the selected apex (e.g., closest in the X-Y coordinate space). In some implementations, the system 100 selects the selected vertex as a vertex corresponding to a center, midpoint, median, or end of the line segment. This may allow the system 100 to reduce polyline segments representing features using the output anchors.In some implementations, the system 100 performs any of various post-processing operations on the output 164 (in which features are represented as polyline segments assigned to output anchors) to determine output data. For example, the system 100 may combine one or more polyline segments or polylines into corresponding features including, but not limited to, lane boundaries, waiting lines, etc.With continued reference to FIG. 1, the system 100 may configure, train, or update at least a subset of the machine learning models 104 (for example, and without limitation, one or more of the encoder 124, the attention processor 128, the attention processors 144, the decoder 152, or the attention processor 156) using training data 180. The training data 180 may be structured in a similar or identical manner to the image data 108 and / or the additional data 120. For example, the training data may include training data instances analogous to the image data 108 (e.g., camera, RADAR, LIDAR, ultrasonic, etc. views of scenes) and labeled with one or more of information analogous to additional data (e.g., position information, poses used to acquire image data, sensor origin and / or beam vector information, etc.), feature labels (e.g., indications of portions of the training data 180, such as pixels representative of lane lines, dividers, or other features), or various combinations thereof. Thus, the system 100 may use the training data 180 to monitor training, configuration, and / or update of one or more of the machine learning models 104.For example, the system 100 may provide one or more training data instances of the training data 180 as input to the encoder 124 (e.g., before or after processing by the feature generator 112, depending on the structure of the training data 180) to cause the encoder 124, the latent array 132, the latent space processing 136, the decoder 152, and / or the output interrogation array 160 (and / or subcomponents thereof) to determine an estimated output. The system 100 may compare the estimated output to the respective training data instance(s) used to determine the estimated output. The system 100 may compare the estimated output to the respective training data instance(s) using any of various comparison functions, for example and without limitation, objective functions or loss functions (e.g., L2loss, L2normals, etc.). The system 100 may use any of various optimization algorithms, such as AdamW optimization, to modify one or more weights or distortions of neural network nodes of one or more machine learning models 104 according to the comparison. This may include, for example and without limitation, modifying the machine learning model(s) 104 until a convergence condition is met (wherein the convergence condition may correspond to at least one of a threshold number of iterations or epochs of configuration or differences between the training data instances and the estimated outputs that are less than a target threshold).FIG. 2 illustrates an example of the output 164 according to some embodiments of the present disclosure. The system 100 may determine the output 164 using one or more operations described with reference to FIG. 1, including various operations of processing image data 108 and / or auxiliary data 120. The system 100 may perform operations including, for example and without limitation, storing the output 164 in any of various database structures, updating the output 164 in response to processing additional image data 108 and / or additional data 120, presenting the output 164 or portions thereof using any of various display devices, combining multiple outputs 164, using the output 164 to identify features in a region of a map according to a location of the region (e.g., for navigation or vehicle controls, etc.), or various combinations thereof.As shown in FIG. 2, the output 160 may have a top-down / BEV perspective extending in X and Y directions (e.g., forward / backward; left / right direction), where Z (e.g., height) position data is assigned to respective features of the output 160. For example, as shown in FIG. 2, the example output 160 represents lane lines 204, 208, 212, 216 and a lane divider 220. The lane lines 204, 208, 212, 216 and the lane divider 220 may be formed as polyline segments (e.g., multiple line segments connected by vertices) and may be assigned to respective selected vertices 206, 210, 214, 218, 222 (each vertex indicating X-Y-coordinates) in the data structure representing the output 160. Features 204, 208, 212, 216, and 220 are shown in dashed lines in FIG. 2 to indicate the curved representations of feature 204, 208, 212, 216, 220 that may be provided by the polyline segments associated with respective vertices 206, 210, 214, 218, 222. For example, in response to querying the output 160 at the location in X-Y space corresponding to the vertex 206, the system 100 may identify the lane line 204, such as to identify the polyline segments that form the lane line 204 in 3D (X-Y-Z) space. As shown in FIG. 2, the output 160 may be formed relative to an origin 202; the origin 202 may correspond to a location (e.g., ego position) of a platform, a sensor, a camera, a vehicle, or various combinations thereof; the origin 202 may be off-center relative to the extents of the output 160 in the X and Y directions, such as to be behind a center of the output 160 relative to the forward / backward X direction.Referring to FIG. 3, each block of the method 300 described herein includes a computing process that may be performed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in a memory. The method 300 may also be embodied as computer usable instructions stored on computer storage media. The method 300 may be provided by a stand-alone application, service or hosted service (stand-alone or in combination with another hosted service), or plug-in for another product, to name just a few. Moreover, method 300 is described by way of example with respect to the systems of FIGS. 1 and 2. However, this method 300 may additionally or alternatively be performed by any system or combination of systems, including, but not limited to, those described herein.FIG. 3 is a flow diagram illustrating a method 300 for real-time multi-view map generation using neural networks in accordance with some embodiments of the present disclosure. Various operations of the method 300 may be implemented by the same or different devices or entities at different times. For example, one or more first devices may implement operations related to configuring (e.g., updating or training) attention neural networks and other machine learning models, and one or more second devices may implement operations related to determining 3D top down / BEV representations of environments and features in environments using image data of the environment from one or more sensors. The one or more second devices may maintain the neural networks or may access the neural networks using APIs provided by the one or more first devices, for example and without limitation.The method 300 includes, at block B 302, receiving sensor images of a scene. The sensor images may be acquired at different times, such as at a sequence of times that includes at least a first time and a second time after the first time (e.g., where the times correspond to one or more steps of an image acquisition rate by the sensor(s)). The sensor images may be captured by the same or different sensors, such as one or more cameras, RADAR sensors, LIDAR sensors, ultrasonic sensors, or combinations thereof, including sensors disposed in different poses (e.g., positions and / or orientations); the sensors may be disposed on a platform, such as a vehicle, or on different platforms. The sensor images may be labeled with information that may be used as inductive priors, such as sensor origin, pose, and / or view direction information. The information may indicate vectors of rays from the origin to different pixels of the sensor images, such as to facilitate position encoding of pixels or groups of pixels. The method may include processing the sensor images to form an input array.In some implementations, the method may include feature generating image data of the sensor images to form the input array. For example, the feature generator may be a convolutional neural network (CNN) to compress the image data into a smaller feature space, such as a latent data space, in which image data (which may be representative of features) is represented as a plurality of tokens. The tokens may correspond to image data of one or more pixels of the sensor images. The tokens may be arranged independently of positions of pixels of the sensor images and assigned position encoding information (e.g., as determined from sensor origin, pose and / or view direction information).The method 300 includes, at block B 304, determining features represented by the sensor images (e.g., by the input array). In some implementations, the method includes determining the features using one or more neural networks, such as attention and / or transformer neural networks. For example, the input device may be provided to an encoder cross-attention network to determine a latent representation of features of the input device, which may be provided to one or more attention networks for the latent space to perform attention operations on the latent representation (e.g., to identify similarities / correspondences between vectors of the latent representation, such as to associate features across vectors of different sensor images), which may be provided to a decoder attention network to perform attention across the latent representation, and an output query device that indicates a structure of an output to be determined (e.g., a grid / map representation, such as a top-down / BEV representation of the scene).The method 300 includes, at block B 306, determining a grid of the scene that associates detected features with cells of the grid. For example, the grid may be a top-down / BEV representation of the scene, such as an X-Y (forward / backward left / right coordinates) representation in which Z (e.g., height) coordinate data is associated with features. For example, the features may be represented as polyline segments, such as a plurality of line segments connected by at least one vertex, such as to represent curved features as Bezier curves using line segments. In some implementations, the method includes selecting an apex of the at least one apex and associating the apex with a cell (e.g., grid point of the grid). The vertex may be selected as a central, median, or final vertex of the plurality of line segments, for example and without limitation. The vertex may be associated with a nearest cell of the grid, such as a cell with a smallest distance in X and / or Y coordinates from the vertex. The grid may be interrogated to identify the features by interrogating the cells to retrieve features associated with the interrogated cells.The method 300 includes at block B 308 at least one of (i) associating the grid with a map data structure or (ii) presenting / rendering / displaying, using a display device, the grid. For example, a map data structure, such as a top-down / BEV map, that shows the line segments that make up the features may be determined by rendering a combined representation of the line segments of each feature. Associating the grid with the map data structure may include performing post-processing operations, such as retrieving, from the grid, one or more output elements (e.g., output anchors), associating features with cells of the grid, identifying corresponding locations in the map data structure, retrieving each of the line segments of the output elements, combining the line segments to form a curve or polyline, and associating the curve or polyline with each pixel of the map data structure corresponding to the curve or polyline (e.g., to form a more complete rendering of the feature in the map data structure). The map data structure may be periodically updated in response to receiving updated sensor images. For example, in response to receiving updated sensor image data (which may correspond to updated locations in a global environment), the at least one neural network may update the grid according to the updated sensor image data.Example Content Streaming SystemReferring to FIG. 4, FIG. 4 is an example system diagram for a content streaming system 400, in accordance with some embodiments of the present disclosure. FIG. 4 includes one or more application servers 402 (which may include similar components, features, and / or functionality to the example computing device 500 of FIG. 5 ), one or more client devices 404 (which may include similar components, features, and / or functionality to the example computing device 500 of FIG. 5 ), and one or more networks 406 (which may be similar to the network(s) described herein). In some embodiments of the present disclosure, system 400 may be implemented to perform diffusion model training and runtime operations. The application session may correspond to a game streaming application (e.g., NVIDIA Ge FORCE NOW), a remote desktop application, a simulation application (e.g., autonomous or semi-autonomous vehicle simulation), computer aided design (CAD) applications, virtual reality (VR) streaming applications, and / or augmented reality (AR) streaming applications, deep learning applications, and / or other types of applications. For example, the system 400 may be implemented to receive inputs indicative of one or more characteristics of an output to be generated using a neural network model, provide the input to the model to cause the model to generate the output, and use the output for various operations, including display or simulation operations.In the system 400, for an application session, the one or more client devices 404 may receive only input data in response to input to the one or more input devices, transmit the input data to the one or more application servers 402, receive encoded display data from the one or more application servers 402, and display the display data on the display 424. Thus, the more compute-intensive compute and processing is offloaded to the one or more application servers 402 (e.g., rendering-particularly ray or path tracking-is performed for a graphical output of the application session by the one or more GPUs of the one or more game servers 402). In other words, the application session is streamed from the one or more application servers 402 to the one or more client devices 404, thereby reducing the requirements of the one or more client devices 404 for graphics processing and rendering.For example, with respect to an instantiation of an application session, a client device 404 may display a frame of the application session on the display 424 based on receiving the display data from the one or more application servers 402. The client device 404 may receive input to one of the one or more input devices and generate input data in response, such as to provide modifying inputs of a drive signal for use by the modifier 112. The client device 404 may transmit the input data to the one or more application servers 402 via the communication interface 420 and via the one or more networks 406 (e.g., the Internet), and the one or more application servers 402 may receive the input data via the communication interface 418. The one or more CPU(s) 408 may receive the input data, process the input data, and transmit data to the one or more GPU(s) 410, causing the one or more GPU(s) 410 to generate a render of the application session. For example, the input data may be representative of movement of a character of the user in a game session of a gaming application, firing of a weapon, reloading, passing a ball, turning a vehicle, etc. The render component 412 may render the application session (e.g., representative of the result of the input data) and the render detect component 414 may detect the render of the application session as display data (e.g., as image data that captures the rendered frame of the application session). Rendering the application session may include ray or path tracked lighting and / or shadow effects calculated using one or more parallel processing units, such as GPUs, which may further use the use of one or more dedicated hardware accelerators or processing cores to perform ray or path tracking techniques, of the one or more application servers 402. In some embodiments, one or more virtual machines (VMs), e.g., including one or more virtual components such as vPUs, vCPUs, etc., may be used by the one or more application servers 402 to support the application sessions. The encoder 416 may then encode the display data to generate encoded display data, and the encoded display data may be transmitted over the one or more networks 406 to the client device 404 via the communication interface 418. The client device 404 may receive the encoded display data via the communication interface 420, and the decoder 422 may decode the encoded display data to generate the display data. The client device 404 may then display the display data via the display 424, such as to display a top-down / BEV map of a scene or environment.Example Computing DeviceFIG. 5 is a block diagram of an example computing device(s) 500 suitable for use in implementing some embodiments of the present disclosure. Computing device 500 may include an interconnect system 502 that directly or indirectly couples the following devices: memory 504, one or more central processing units (CPUs) 506, one or more graphics processing units (GPUs) 508, a communication interface 510, input / output (I / O) ports 512, input / output components 514, a power supply 516, one or more presentation components 518 (e.g., display(s)), and one or more logic units 520. In at least one embodiment, computing device(s) 500 may include one or more virtual machines (VMs), and / or any of components thereof may include virtual components (e.g., virtual hardware components). For non-limiting examples, one or more of the GPUs 508 may include one or more vPUs, one or more of the CPUs 506 may include one or more vCPUs, and / or one or more of the logic units 520 may include one or more virtual logic units. Thus, a computing device(s) 500 may include discrete components (e.g., a full GPU dedicated to computing device 500), virtual components (e.g., a portion of a GPU dedicated to computing device 500), or a combination thereof.Although the various blocks of FIG. 5 are shown as being connected to lines via the connection system 502, this is not intended to be limiting and is for clarity only. For example, in some embodiments, a presentation component 518, such as a display device, may be considered an I / O component 514 (e.g., when the display is a touch screen). As another example, the CPUs 506 and / or GPUs 508 may include memory (e.g., the memory 504 may be representative of a storage device in addition to the memory of the GPUs 508, the CPUs 506, and / or other components). In other words, the computing device of FIG. 5 is illustrative only. Categories such as "workstation," "server," "laptop," "desktop," "tablet," "client device," "mobile device," "handheld device," "game console," "electronic control unit (ECU)," "virtual reality system," and / or other device or system types are not distinguished, as all are contemplated within the scope of the computing device of FIG. 5.The interconnect system 502 may represent one or more interconnects or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnect system 502 may be arranged in various topologies including, but not limited to, bus, star, ring, mesh, tree, or hybrid topologies. The interconnect system 502 may include one or more types of bus or interconnects, such as an industry standard architecture (ISA) bus, an extended industry standard architecture (EISA) bus, a video electronics standards association (VESA) bus, a peripheral component interconnect (PCI) bus, a peripheral component interconnect (PCIe) bus, and / or another type of bus or interconnect. In some embodiments, there are direct connections between components. As an example, the CPU 506 may be directly connected to the memory 504. Further, the CPU 506 may be directly connected to the GPU 508. If there is a direct or point-to-point connection between components, the interconnect system 502 may include a PCIe connection to perform the connection. In these examples, a PCI bus need not be included in computing device 500.The memory 504 may include any of a variety of computer readable media. The computer readable media may be any available media that can be accessed by the computing device 500. The computer readable media may include both volatile and non-volatile media, and removable and non-removable media. By way of example, and not limitation, the computer readable media may include computer storage media and communication media.The computer storage media may include both volatile and non-volatile media and / or removable media and non-removable media implemented in any method or technology for storing information such as computer readable instructions, data structures, program modules, and / or other types of data. For example, the memory 504 may store computer readable instructions (e.g., representing a program or programs and / or a program element or elements), such as an operating system. Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVDs) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by the computing device 500. As used herein, computer storage media does not include signals per se.The computer storage media may embody computer readable instructions, data structures, program modules, and / or other types of data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" may refer to a signal in which one or more of its characteristics are adjusted or altered in such a way that information is encoded in the signal. By way of example, and not limitation, the computer storage media may include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of any of the foregoing should also be included within the scope of computer readable media.The CPU(s) 506 may be configured to execute at least some of the computer readable instructions to control one or more components of the computing device 500 to perform one or more of the methods and / or processes described herein. The CPU(s) 506 may each include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of handling a plurality of software threads simultaneously. The CPU(s) 506 may include any type of processor and may include different types of processors, depending on the type of computing device 500 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 500, the processor may be an Advanced RISC Machines (ARM) processor implemented using Reduced Instruction Set Computing (RISC), or an x86 processor implemented using Complex Instruction Set Computing (CISC). Computing device 500 may include one or more CPUs 506 in addition to one or more microprocessors or supplemental coprocessors, such as mathematical coprocessors.Additionally or alternatively to the CPU(s) 506, the GPU(s) 508 may be configured to execute at least some of the computer readable instructions to control one or more components of the computing device 500 to perform one or more of the methods and / or processes described herein. One or more of the GPU(s) 508 may be an integrated GPU (e.g., with one or more of the CPU(s) 506 and / or one or more of the GPU(s) 508 may be a discrete GPU. In embodiments, one or more of the GPU(s) 508 may be a coprocessor of one or more of the CPU(s) 506. The GPU(s) 508 may be used by the computing device 500 to render graphics (e.g., 3D graphics) or perform general purpose computations. For example, the GPU(s) 508 may be used for universal computing on GPUs (GPGPU). The GPU(s) 508 may include hundreds or thousands of cores capable of handling hundreds or thousands of software threads simultaneously. The GPU(s) 508 may generate pixel data for output images in response to render commands (e.g., render commands from the one or more CPU(s) 506 received via a host interface). The GPU(s) 508 may include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. The display memory may be included as part of the memory 504. The GPU(s) 508 may include two or more GPUs operating in parallel (e.g., via a link). The connection may connect the GPUs directly (e.g., using NVLINK) or may connect the GPUs through a switch (e.g., using NVSwit). When combined, each GPU 508 may generate pixel data or GPGPU data for different portions of an output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory or may share memory with other GPUs.Additionally or alternatively to the CPU(s) 506 and / or the GPU(s) 508, the logic unit(s) 520 may be configured to execute at least some of the computer readable instructions to control one or more components of the computing device 500 to perform one or more of the methods and / or processes described herein. In embodiments, the CPU(s) 506, the GPU(s) 508, and / or the logic unit(s) 520 may discretely or jointly perform any combination of the methods, processes, and / or portions thereof. One or more of the logic units 520 may be part of and / or integrated with one or more of the CPU(s) 506 and / or the GPU(s) 508 and / or one or more of the logic units 520 may be discrete components or otherwise external to the CPU(s) 506 and / or the GPU(s) 508. In embodiments, one or more of the logic units 520 may be a coprocessor of one or more of the CPU(s) 506 and / or one or more of the GPU(s) 508.Examples of logic unit(s) 520 include one or more processing cores and / or components thereof, such as computing units (DPUs), tensor cores (TCs), tensor processing units (TPUs), visual pixel cores (PVCs), vision processing units (VPUs), image processing units (IPUs), graphics processing clusters (GPCs), texture processing clusters (TPCs), streaming multiprocessors (SMs), tree traversal units (TTUs), artificial intelligence accelerators (AIAs), deep learning accelerators (DLAs), arithmetic logic units (ALUs), application specific integrated circuits (ASICs), Floating point units (FPUs), input / output (I / O) elements, peripheral component interconnect (PCI) or peripheral component interconnect express (PCIe) elements, and / or the like.Communication interface 510 may include one or more receivers, transmitters, and / or transceivers that enable computing device 500 to communicate with other computing devices over an electronic communication network, including wired and / or wireless communication. The communication interface 510 may include components and functionality to enable communication over any of a number of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communicating over Ethernet or InfiniBand), low-power wide area networks (e.g., LoRaWA, SigOx, etc.), and / or the Internet. In one or more embodiments, the logic unit(s) 520 and / or the communication interface 510 may include one or more computing devices (DPUs) to directly transmit data received over a network and / or through the connection system 502 to (e.g., a memory of) one or more GPU(s) 508. In some embodiments, a plurality of computing devices 500 or components thereof, which may be similar or different from each other in various respects, may be communicatively coupled to transmit and receive data for performing various operations described herein, such as to facilitate latency reduction.I / O ports 512 may allow computing device 500 to be logically coupled to other devices, including I / O components 514, presentation component(s) 518, and / or other components, some of which may be incorporated (e.g., integrated) into computing device 500. Illustrative I / O components 514 include a microphone, a mouse, a keyboard, a joystick, a game pad, a game controller, a satellite dish, a scanner, a printer, a wireless device, etc. The I / O components 514 may provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by a user, such as to generate a drive signal for use by the modifier 112 or a reference image (e.g., images 104). In some cases, input may be transmitted to a suitable network element for further processing, such as to modify and register images. A NUI may implement any combination of speech recognition, stylus recognition, face recognition, biometric recognition, gesture recognition on both the screen and adjacent to the screen, air gestures, head and eye tracking, and touch recognition (as described in more detail below) associated with a display of computing device 500. The computing device 500 may include depth cameras, such as stereoscopic camera systems, infrared camera systems, RGB camera systems, touch screen technology, and combinations thereof, for gesture sensing and recognition. Additionally, computing device 500 may include accelerometers or gyroscopes (e.g., as part of an inertial measurement unit (IMU)) that enable detection of motion. In some examples, the output of the accelerometers or gyroscopes may be used by computing device 500 to render immersive augmented reality or virtual reality.The power supply 516 may include a hard-wired power supply, a battery power supply, or a combination thereof. The power supply 516 may provide power to the computing device 500 to enable the components of the computing device 500 to operate.The presentation component(s) 518 may include a display (e.g., a monitor, a touch screen, a television screen, a head-up display (HUD), other types of display, or a combination thereof), speakers, and / or other presentation components. The presentation component(s) 518 may receive data from other components (e.g., the one or more GPU(s) 508, the one or more CPU(s) 506, DPUs, etc.) and output the data (e.g., as an image, video, sound, etc.).EXAMPLE DATACENTERFIG. 6 illustrates an example data center 600 that may be used in at least one embodiment of the present disclosure, such as to implement systems 100, 200 in one or more examples of data center 600. The data center 600 may include a data center infrastructure layer 610, a framework layer 620, a software layer 630, and / or an application layer 640.As shown in FIG. 6, data center infrastructure layer 610 may include resource orchestrator 612, grouped compute resources 614, and node compute resources ("node C.R.s") 616( 1)- 616(N), where "N" represents any integer, positive integer. In at least one embodiment, node C.R.s 616( 1)- 616(N) may include, but are not limited to, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), storage devices (e.g., dynamic read-only memories), storage devices (e.g., solid state or disk drives), network input / output (NW I / O) devices, network switches, virtual machines (VMs), power modules, and / or cooling modules, etc. In some embodiments, one or more node C.R.s from the node C.R.s 616( 1)- 616(N) may correspond to a server having one or more of the computing resources mentioned above. Moreover, in some embodiments, the node C.R.s 616( 1)- 616(N) may include one or more virtual components, such as vPUs, vCPUs, and / or the like, and / or one or more of the node C.R.s 616( 1)- 616(N) may correspond to a virtual machine (VM).In at least one embodiment, grouped computing resources 614 may include separate groupings of node C.R.s 616 residing in one or more racks (not shown) or multiple racks residing in data centers at different geographic locations (also not shown). Separate groupings of node C.R.s 616 within grouped computing resources 614 may include grouped computing, network, storage, or storage resources that may be configured or allocated to support one or more workloads. In at least one embodiment, multiple node C.R.s 616, including CPUs, GPUs, DPUs, and / or other processors, may be grouped into one or more racks to provide computing resources to support one or more workloads. The one or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.The resource orchestrator 612 may configure or otherwise control one or more node C.R.s 616( 1)- 616(N) and / or grouped computing resources 614. In at least one embodiment, resource orchestrator 612 may include a software design infrastructure (SDI) management entity for data center 600. The resource orchestrator 612 may include hardware, software, or a combination thereof.In at least one embodiment, as shown in FIG. 6, framework layer 620 may include a job scheduler 628, a configuration manager 634, a resource manager 636, and / or a distributed file system 638. The framework layer 620 may include a framework to support software 632 of the software layer 630 and / or one or more application(s) 642 of the application layer 640. The software 632 or application(s) 642 may include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. The framework layer 620 may be, among other things, a type of free and open source software web application framework, such as Apache Spark™ (hereafter "Spark"), which may use the distributed file system 638 for large-scale data processing (e.g., "Big Data"). In at least one embodiment, job scheduler 628 may include a spark driver to facilitate scheduling workloads supported by different layers of data center 600. The configuration manager 634 may be able to configure various layers, such as the software layer 630 and the framework layer 620, including spark and the distributed file system 638, to support large-scale data processing. The resource manager 636 may be able to manage clustered or grouped computing resources mapped to or assigned to support the distributed file system 638 and the job scheduler 628. In at least one embodiment, clustered or grouped computing resources may include grouped computing resource 614 at data center infrastructure layer 610. The resource manager 636 may coordinate with the resource orchestrator 612 to manage these mapped or allocated computing resources.In at least one embodiment, software 632 included in software layer 630 may include software used by at least portions of node C.R.s 616( 1)- 616(N), grouped computing resources 614, and / or distributed file system 638 of framework layer 620. One or more types of software may include, but are not limited to, Internet web page search software, email virus scan software, database software, and streaming video content software.In at least one embodiment, application(s) 642 included in application layer 640 may include one or more types of applications used by at least portions of node C.R.s 616( 1)- 616(N), grouped computing resources 614, and / or distributed file system 638 of framework layer 620. One or more types of applications may include, but are not limited to, any number of a genomic application, a cognitive computing application, and a machine learning application, including training or inferencing software, machine learning framework software (e.g., PyTor, TensorFlow, Coffee, etc.), and / or other machine learning applications used in connection with one or more embodiments, such as for training, configuring, updating, and / or executing machine learning models 104.In at least one embodiment, any of configuration manager 634, resource manager 636, and resource orchestrator 612 may implement any number and type of self-modifying actions based on any amount and type of data captured in any technically possible manner. Self-modifying actions may relieve a data center operator of data center 600 from making potentially bad configuration decisions and avoiding potentially underutilized and / or poorly executing portions of a data center.The data center 600 may include tools, services, software, or other resources to train one or more machine learning models (e.g., training models 104, 204 and / or neural networks 106, 206, etc.) or predict or derive information using one or more machine learning models, according to one or more embodiments described herein. For example, a / e machine learning model(s) may be trained by computing weight parameters according to a neural network architecture using software and / or computing resources described above with respect to the data center 600. In at least one embodiment, trained or deployed machine learning models corresponding to one or more neural networks may be used to derive or predict information using resources described above with respect to data center 600 by using weight parameters calculated by one or more training techniques, such as, but not limited to, those described herein.In at least one embodiment, data center 600 may use CPUs, application specific integrated circuits (ASICs), GPUs, FPGAs, and / or other hardware (or virtual computing resources corresponding thereto) to perform training and / or inference using resources described above. Moreover, one or more software and / or hardware resources described above may be configured as a service to enable users to train information or perform inference therefrom, such as image recognition, speech recognition, or other artificial intelligence services.EXAMPLE NETWORK ENVIRONMENTSNetwork environments suitable for use in implementing embodiments of the disclosure may include one or more client devices, servers, network attached storage (NAS), other backend devices, and / or other device types. The client devices, servers, and / or other device types (e.g., each device) may be implemented on one or more instances of the one or more computing device(s) 500 of FIG. 5 -e.g., each device may include similar components, features, and / or functionality of the one or more computing device(s) 500. Moreover, when backend devices (e.g., servers, NAS, etc.) are implemented, the backend devices may be included as part of a data center 600, an example of which is described in more detail herein with respect to FIG. 6.Components of a network environment may communicate with each other over one or more networks, which may be wired, wireless, or both. The network may include multiple networks or a network of networks. For example, the network may include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks such as the Internet and / or a public switched telephone network (PSTN), and / or one or more private networks. When the network includes a wireless telecommunications network, components such as a base station, a communication tower, or even access points (as well as other components) may provide wireless connectivity.Compatible network environments may include one or more peer-to-peer network environments - in which case a server may not be included in a network environment - and one or more client-server network environments - in which case one or more servers may be included in a network environment. In peer-to-peer network environments, the functionality described herein with respect to one or more servers may be implemented on any number of client devices.In at least one embodiment, a network environment may include one or more cloud-based network environments, a distributed computing environment, a combination thereof, etc. A cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more of servers, which may include one or more core network servers and / or edge servers. A framework layer may include a framework to support software of a software layer and / or one or more application(s) of an application layer. The software or application(s) may each include web-based service software or applications. In embodiments, one or more of the client devices may use the web-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, among other things, some type of free and open source software web application framework, such as may use a distributed file system for large-scale data processing (e.g., "big data").A cloud-based networking environment may provide cloud computing and / or cloud storage that performs any combination of computing and / or storage functions described herein (or one or more portions thereof). Any of these various functions may be distributed across multiple locations of central or core servers (e.g., one or more data centers, which may be distributed across a state, region, country, globe, etc.). If a connection to a user (e.g., a client device) is relatively close to one or more edge servers, one or more core servers may designate at least a portion of the functionality for the edge server or servers. A cloud-based network environment may be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).The one or more client devices may include at least some of the components, features, and functionality of the example computing device(s) 500 described herein with respect to FIG. 5. For example, and not by way of limitation, a client device may be used as a personal computer (PC), a laptop computer, a mobile device, a smart phone, a tablet computer, a smart watch, a portable computer, a personal digital assistant (PDA), an MP3 player, a virtual reality headset, a global positioning system (GPS) device or device, a video player, a video camera, a monitoring device or system, a vehicle, a boat, a flying ship, a virtual machine, a drone, a robot, a hand held communication device, a hospital device, a gaming device or system, an entertainment system, a vehicle computer system, an embedded system controller, and the like, a remote control, a device, an entertainment electronics device, a workstation, an edge device, any combination of these delimited devices, or any other suitable device.The disclosure may be described in the general context of computer code or machine-usable instructions, including computer-executable instructions, such as program modules, executed by a computer or other machine, such as a personal data assistant or other handheld device. Generally, program modules, including routines, programs, objects, components, data structures, etc., refer to code that performs certain tasks or implements certain abstract data types. The disclosure may be practiced in a variety of system configurations, including handheld devices, consumer electronics, general purpose computers, specialized computing devices, etc. The disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network.As used herein, a recitation of "and / or" with respect to two or more elements should be interpreted to mean only one element or a combination of elements. For example, "element A, element B, and / or element C" may include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. Moreover, "at least one of element A or element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Further, "at least one of element A and element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.The subject matter of the present disclosure is specifically described herein to meet legal requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors have contemplated that the claimed subject matter could be embodied in other ways to include various steps or combinations of steps similar to those described herein in connection with other current or future technologies. Moreover, although the terms "step" and / or "block" may be used herein to refer to various elements of methods used, the terms should not be interpreted as implying a particular order between or between various steps disclosed herein unless and unless the order of individual steps is explicitly described.

Claims

A processor comprising: one or more circuits to: receive a first sensor image captured at a first time and a second sensor image captured at a second time; determine, using a neural network and based at least on the first sensor image and the second sensor image, one or more features represented by the first sensor image and the second sensor image; determine, using the neural network, a grid of the scene in which the one or more features are each associated with a cell of the grid; and at least one of (i) associating the grid with a map data structure or (ii) representing the grid using a display device.The processor of claim 1, wherein the one or more circuits are to determine the one or more features using at least one of radio detection and ranging (RADAR) data, light detection and ranging (LIDAR) data, or ultrasound data corresponding to at least one of the first sensor image or the second sensor image.The processor of claim 1, wherein the one or more circuits are to provide, as input to the neural network, a position representation of at least one of a camera center of the first sensor image, a camera center of the second sensor image, a vector of a ray to a feature of the first sensor image, or a vector of a ray to a feature of the second sensor image.The processor of claim 1, wherein the neural network comprises: a feature generator to convert image data of the first sensor image and the second sensor image into a plurality of tokens in a latent data space; an encoder cross-attention processor to process the plurality of tokens and a latent data representation maintained by one or more self-attention modules; and a decoder cross-attention processor to process an intermediate output of the neural network and the latent data representation to determine the grid of the scene.The processor of claim 1, wherein the grid comprises a two-dimensional representation of the scene in a top-down reference frame, and the one or more circuits are to determine, for each feature of the one or more features, a polyline representing the feature, the polyline comprising a plurality of points indicating a plurality of line segments, the one or more circuits being to associate the feature with the cell by associating the polyline with the cell.The processor of claim 1, wherein the one or more circuits are to associate a height of the feature and / or a class of the feature with the cell.The processor of claim 1, wherein the first sensor image and the second sensor image comprise camera data.The processor of claim 1, wherein the processor is included in at least one of: an autonomous or semi-autonomous machine control system; an autonomous or semi-autonomous machine perception system; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing conversation AI operations; a system for generating synthetic data; a system including one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.A processor, comprising: one or more circuits to: receive training data comprising a first sensor image captured at a first time and a second sensor image captured at a second time, wherein at least one feature is associated with the first sensor image and the second image; determine, using at least one neural network and based at least on the first sensor image and the second sensor image, an estimated output indicative of at least one position of at least one estimated feature; and update the at least one neural network based at least on the estimated output and the at least one feature.The processor of claim 9, wherein the first sensor image and the second sensor image comprise at least one of camera data, radio detection and ranging (RADAR) data, light detection and ranging (LIDAR) data, or ultrasound data.The processor of claim 9, wherein the at least one feature comprises at least one of a lane line, a lane divider, a waiting line, or a boundary structure.The processor of claim 1, wherein the at least one neural network comprises an encoder attention network, a plurality of latent space attention networks, and a decoder attention network.The processor of claim 12, wherein the processor is included in at least one of: an autonomous or semi-autonomous machine control system; an autonomous or semi-autonomous machine perception system; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing conversation AI operations; a system for generating synthetic data; a system including one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.A method comprising: receiving, using one or more processors, a first sensor image captured at a first time and a second sensor image captured at a second time; determining, using the one or more processors and a neural network and based at least on the first sensor image and the second sensor image, one or more features represented by the first sensor image and the second sensor image; determining, using the one or more processors and the neural network, a grid of the scene in which the one or more features are each associated with a cell of the grid; and at least one of (i) associating, using the one or more processors, the grid with a map data structure, or (ii) presenting, using the one or more processors and a display device, the grid.The method of claim 14, further comprising determining, using the one or more processors, the one or more features using at least one of radio detection and ranging (RADAR) data, light detection and ranging (LIDAR) data, or ultrasound data corresponding to at least one of the first sensor image or the second sensor image.The method of claim 14, further comprising providing, using the one or more processors, as input to the neural network, a position representation of at least one of a camera center of the first sensor image, a camera center of the second sensor image, a vector of a ray to a feature of the first sensor image, or a vector of a ray to a feature of the second sensor image.The method of claim 14, wherein the neural network comprises: a feature generator for converting image data of the first sensor image and the second sensor image into a plurality of tokens in a latent data space; an encoder cross-attention processor for processing the plurality of tokens and a latent data representation maintained by one or more self-attention modules; and a decoder cross-attention processor for processing an intermediate output of the neural network and the latent data representation to determine the grid of the scene.The method of claim 14, wherein the grid comprises a two-dimensional representation of the scene in a top-down reference frame, and the method further comprises determining, using the one or more processors, for each feature of the one or more features, a polyline representing the feature, the polyline comprising a plurality of points indicative of a plurality of line segments, the one or more circuits to associate the feature with the cell by associating the polyline with the cell.The method of claim 14, further comprising associating, using the one or more processors, a height of the feature and / or a class of the feature with the cell.The method of claim 14, wherein the first sensor image and the second sensor image comprise camera data.

Images