Feature Recognition Models for Autonomous and Semi-Autonomous Systems and Applications

Abstract

This document describes various examples of feature recognition models for autonomous and / or semi-autonomous systems and applications. The systems and methods described here can use one or more trained machine learning models to automatically generate map-like representations of traffic features, such as road markings and / or road edges. The one or more models can, for example, take as input an image depicting at least a section of a map showing one or more traffic features along with one or more references to points associated with those traffic features, as represented by the image.Based at least on the processing of the inputs, one or more models can generate and / or output data representing additional points associated with one or more traffic features, and / or a heatmap representing one or more lines depicting one or more traffic features. This output data can then be used to determine the one or more representations of the one or more traffic features for annotating the map.

Description

BACKGROUND

[0001] For vehicles (e.g., autonomous vehicles, semi-autonomous vehicles, robots, etc.) to operate safely in environments, they must be able to effectively perform vehicle maneuvers, such as lane keeping, lane changes, lane splitting, turning, stopping and starting at intersections, pedestrian crossings, and the like, and / or other vehicle or machine maneuvers. For example, for a vehicle to navigate roads on the same level as its surroundings (e.g., urban streets, side streets, neighborhood streets, etc.) and on highways (e.g., multi-lane roads), it must be able to navigate between one or more subdivisions or boundaries (e.g., lanes, intersections, pedestrian crossings, barriers, etc.) of a roadway, which are often marked using traffic features such as road markings containing arrows, text, graphics, and / or other types of content.Therefore, it is important that the vehicles can recognize the traffic characteristics within the environments so that the vehicles can determine how to navigate according to the rules assigned to the traffic characteristics.

[0002] To detect traffic features, vehicles can use maps, at least in part, that correspond to the environments in which they navigate. For example, maps can be annotated to indicate the locations of important traffic features that vehicles need to identify while navigating, such as road edges, road markings, traffic signs, and so on. Some conventional approaches to annotating such maps involve users viewing different sections of the map to manually enter labels for the traffic features. For example, a user might manually specify the location of a road marking by selecting a number of points along the road marking, such as hundreds and / or thousands of points, for a given length of the road marking.However, requiring users to manually specify the locations of traffic features, as represented by maps, can be time-consuming, prone to user error, and / or require a large amount of computing resources (from various user devices).

[0003] Therefore, and particularly for road markings, other conventional approaches can use curve fitting functions to connect existing road markings that are already annotated on maps. For example, if users have already annotated a first segment of a road marking and a separate, second segment of the road marking, these conventional approaches will simply join the two segments of the road marking using a curve fitting function. However, by simply using curve fitting functions to connect existing road markings, these conventional approaches may be accurate with respect to a straight road marking, but inaccurate for road markings that contain one or more curves.Since these conventional approaches also operate on the entirety of a map, the generated annotations for the road markings cannot align if the map is segmented into subsections (e.g., images), such as to provide the map for vehicles to navigate. SUMMARY

[0004] The invention is defined by the claims. To illustrate the invention, aspects and embodiments that may or may not fall within the scope of protection of the claims are described herein.

[0005] Embodiments of the present disclosure relate to feature recognition models for autonomous or semi-autonomous systems and applications. The systems and methods described herein can use one or more trained machine learning models (the one or the models) to automatically generate representations of traffic features corresponding to a map, such as road markings and / or road edges. The one or the models can, for example, take as input an image representing at least a portion of a map containing one or more traffic features along with one or more references to one or more points (e.g., one or more prompts) associated with the one or more traffic features as represented by the image.Based at least on the processing of the inputs, one or more models can generate and / or output data representing additional points associated with one or more traffic features, and / or a heatmap representing one or more lines corresponding to one or more traffic features. In some examples, one or more models and / or another post-processing component can then determine one or more final representations for one or more traffic features using the outputs, such as line representations for road markings and / or road edges, which can then be used to annotate the map.

[0006] In contrast to conventional systems, the systems of this disclosure, in some embodiments, are able to use the prompts and / or the input images to automatically determine the positions of traffic features as represented by maps. In this way, the systems of this disclosure do not require users to manually enter all the points for the traffic features, such as hundreds and / or thousands of points, when annotating the maps. Additionally, and as described in more detail herein, one or more models can be trained to determine a number of points associated with the traffic features, such as up to one hundred or more points, which are then used to determine the final representations (e.g., line representations) for the traffic features within the map.Thus, the systems of the present disclosure can generate more accurate traffic feature annotations compared to previous systems that merely use line matching to connect lines already labeled for the maps.

[0007] Further features of the disclosure are characterized by the independent and dependent claims.

[0008] Any feature of one aspect of the disclosure can be applied in any suitable combination to other aspects of the disclosure. In particular, procedural aspects can be applied to apparatus or system aspects, and vice versa.

[0009] Furthermore, features implemented in hardware can be implemented in software and vice versa. Any reference to software and hardware features here should be interpreted accordingly.

[0010] Each system or device feature described here can also be provided as a process feature, and vice versa. System and / or device aspects that are functionally described (including means plus functional features) can alternatively be expressed in terms of their corresponding structure, such as a suitably programmed processor and associated working memory.

[0011] It is also understood that certain combinations of the various features described and defined in each aspect of the revelation can be implemented and / or provided and / or used independently of one another.

[0012] The disclosure also provides computer programs and computer program products comprising software code designed to perform one of the methods described herein when executed on a data processing device and / or to embody one of the device and system features described herein, including one or all component steps of a method.

[0013] The disclosure also includes a computer or computing system (including networked or distributed systems) with an operating system that supports a computer program for carrying out the procedures described herein and / or for embodying the device or system features described herein.

[0014] The disclosure also provides a computer-readable medium on which one or more of the aforementioned computer programs are stored.

[0015] The revelation also provides a signal that carries one or more of the aforementioned computer programs.

[0016] The disclosure extends to methods and / or devices and / or systems as described herein with reference to the accompanying drawings.

[0017] Aspects and embodiments of the disclosure will now be described purely by way of example with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The present systems and methods for feature recognition models for autonomous and / or semi-autonomous systems and applications are described in detail below with reference to the accompanying drawings. These show: Fig. 1A an exemplary data flow diagram for a process for feature recognition and / or annotation of a map using one or more machine learning models, according to some embodiments of the present disclosure; Fig. 1B an example of an architecture of one or more machine learning models that recognize features, according to some embodiments of the present disclosure; Fig. 1C Examples of cross-attention layers that can be used by one or more machine learning models, according to some embodiments of the present disclosure; Fig. 1D an example of one or more machine learning models that output information associated with points of a feature, according to some embodiments of the present disclosure; Fig. 2A-2B an example of segmenting a map to generate images that represent features within an environment, according to some embodiments of the present disclosure; Fig. 3 an example of determining positions of input points that are associated with features represented by an image, according to some embodiments of the present disclosure; Fig. 4 an example of determining points that are associated with features represented by an image, according to some embodiments of the present disclosure; Fig. 5 an example of a heatmap indicating the positions of features as shown in an image, according to some embodiments of the present disclosure; Fig. 6 an example of annotating a map to contain information associated with features, according to some embodiments of the present disclosure; Fig. 7 a data flow diagram illustrating a process for training one or more machine learning models to recognize features associated with images and / or maps, according to some embodiments of the present disclosure; Fig. 8 an example of one or more systems that can be configured to perform at least part of the process described herein, according to some embodiments of the present disclosure; Fig. 9-10 Flowcharts showing methods for recognizing features associated with images, according to some embodiments of the present disclosure; Fig. 11A an illustration of an exemplary autonomous vehicle, according to some embodiments of the present disclosure; Fig. 11B is an example of camera locations and fields of view for the exemplary autonomous vehicle from Fig. 11A, according to some embodiments of the present disclosure; Fig. 11C a block diagram of an exemplary system architecture for the exemplary autonomous vehicle from Fig. 11A, according to some embodiments of the present disclosure; Fig. 11D a system diagram for the communication between one or more cloud-based servers and the example autonomous vehicle from Fig. 11A, according to some embodiments of the present disclosure; Fig. 12 a block diagram of an exemplary computing device suitable for use in the implementation of some embodiments of the present disclosure; and Fig. 13 a block diagram of an exemplary data center suitable for use in the implementation of some embodiments of the present disclosure. DETAILED DESCRIPTION

[0019] Systems and methods relating to feature recognition models for autonomous and semi-autonomous systems and applications are disclosed. Although the present disclosure relates to an exemplary autonomous or semi-autonomous vehicle or an exemplary autonomous or semi-autonomous machine 1100 (here alternatively referred to as "Vehicle 1100", "Ego-Vehicle 1100", "Ego-Machine 1100" or "Machine 1100"), of which an example is given in relation to Fig. The fact that the systems and procedures described in sections 11A-11D can be described is not a limiting factor. The systems and procedures described here can be used, for example, without restriction, by non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, steered and unsteered robots or robotic platforms, warehouse vehicles, all-terrain vehicles, vehicles coupled with one or more trailers, hydrofoils, boats, shuttle vehicles, emergency vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, underwater vehicles, drones, and / or other types of vehicles.Although the present disclosure is described in relation to the recognition of traffic features and / or the annotation of maps with labels that are associated with traffic features, this is not to be understood as a limitation beyond this, and the systems and methods described herein can be used in augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications and / or any other technology fields where object or feature recognition and / or map creation can be used.

[0020] One or more systems can, for example, generate, receive, retrieve, obtain, and / or store map data corresponding to a map, such as a navigation map, a standard definition (SD) map, a high definition (HD) map, an occupancy map or raster, and / or any other type of map associated with an environment. In some examples, the map may be associated with different layers corresponding to different types of sensor modalities and / or different types of features. For example, the map may be associated with one or more image layers, one or more LiDAR layers, one or more radar layers, one or more standby state layers, one or more road marking layers, and / or any other type of layer.Additionally, the map can display information associated with features located within the environment, including but not limited to feature positions, feature orientations, feature classifications, and / or any other information. As described here, the features can include traffic features such as road edges, road markings, carriageway edges, lane markings, traffic poles, traffic signs, traffic lights, and / or the like, and / or any other type of feature and / or object that may be located in the environment.

[0021] The one or more systems can then perform various techniques to generate annotations for at least some of the features represented by the map. For example, the one or more systems can segment the map into sections representing specific areas of the environment. As described here, in some examples, the map sections can be represented using images, such as top-down images (e.g., bird's-eye view images) and / or intensity images, that depict the different areas of the environment. The LiDAR data (e.g., the LiDAR layer) associated with the map and / or point cloud data associated with the LiDAR data can, for example, represent the intensities of points within the environment.The LiDAR sensors used to generate LiDAR data can, for example, measure the intensities of points at the moment light returns to them. These intensities can be represented numerically, such as numbers between 0 and 256 (and / or any other range), where the numbers vary based on the composition (e.g., color, texture, material, etc.) of the surface from which the light is reflected. A low number might indicate low reflectivity, while a high number might indicate high reflectivity. In some cases, the intensities may depend on other factors, such as the angle of incidence, the areas of the points, and / or similar conditions.

[0022] Thus, one or more systems can process the LiDAR data and / or the point cloud data to generate the images (e.g., the top-down intensity images) that are mapped. Since the intensities of the points can vary based on one or more factors, such as the colors of the surfaces associated with the points (e.g., the colors of the surfaces for which light is reflected), the images can indicate the structure of several features located within the environment. For example, if a feature includes a road marking located on a road surface within the environment, with the road marking painted using a specific color (e.g., white, yellow, etc.), then an image of the environment can show the points associated with the road marking as a different color compared to the points associated with other features, such as...depicting the road surface itself.

[0023] The one or more systems can then use the images to recognize the features located within the environment. For an image, the one or more systems can, for example, determine one or more points (the one or more prompt points) associated with a feature (such as a lane marking) as depicted in the image. In some examples, the one or more systems can determine the one or more prompt points using input data representing one or more inputs from one or more users specifying the one or more positions of the one or more prompt points in the image. Additionally or alternatively, in some examples, the one or more systems can process the image and / or map to automatically determine the one or more positions of the one or more prompt points in the image.A section of the feature, as represented by the map, may already have been annotated, for example, using one or more of the processes described here. Thus, one or more systems can use the annotations to determine the one or more positions of the one or more prompt points in the image.

[0024] The one or more systems can then process input data associated with the image and / or the one or more prompt points using one or more machine learning models (e.g., the one or more models) trained to determine information associated with the feature, such as points (e.g., the output points) and / or a representation of the feature. The one or more models may, for example, include one or more encoders (the one or more image encoders) configured to generate one or more image embeds associated with the image. The one or more models may also include one or more additional encoders (the one or more prompt encoders) configured to generate one or more tokens associated with the one or more prompts.In some examples, the one or more prompt encoders may be configured to generate a token associated with each of the one or more prompt points. The one or more models may append the one or more tokens to one or more learnable tokens in some examples to generate one or more input tokens, as described in more detail here. However, and as described in more detail here, in some examples the one or more systems may not receive any prompt points associated with the image. In such examples, the one or more input tokens may only contain the one or more learnable tokens.

[0025] The one or more models can then contain one or more cross-attention components configured to process the one or more image embeddings and the one or more input tokens. In some examples, the one or more cross-attention components can include one or more cross-attention layers, such as one or more two-sided cross-attention layers, one or more token-image transformers, and / or any other type of processing component configured to perform one or more of the processes described herein. Based on at least the processing, the one or more cross-attention components can include at least one token (e.g., the one or more output tokens) associated with one or more positions of the one or more prompt points, and at least one token (e.g.,Generate and / or output the one or more image tokens associated with the image. In some examples, the one or more output tokens may be associated with a vector containing a specific dimension.

[0026] The one or more models can then process the one or more output tokens and / or the image token using one or more decoders configured to determine output points associated with the feature. In some examples, the one or more decoders can be any type of decoder, such as an autoregressive transformer decoder. Additionally, in some examples, the one or more decoders can be configured to determine a threshold (e.g., 100) of output points using a sequence.The one or more decoders can, for example, determine a first output point using the one or more output tokens, a second output point using a second output token associated with the first output point (and / or any other preceding output token), a third output point using a third output token associated with the second output point (and / or any other preceding output token), a fourth output point using a fourth input token associated with the third output point (and / or any other preceding output token), and / or so on until the threshold of points is reached.

[0027] In some examples, the one or more decoders determine the additional output tokens using a matrix (which can represent one or more learnable tokens) containing dimensions that correspond to the size of the image. For example, if the image size is 1024*512, then the dimensions of the matrix can be 1024 units in the x-direction and 512 units in the y-direction. To determine an output token, the one or more decoders can use the matrix and the previous output point. For example, to determine the second output token in the example above, the one or more decoders can use the matrix and the first output point to determine a vector associated with the first output point, with the vector corresponding to the second output token.Additionally, one or more decoders can use the matrix and the second output point to determine a vector associated with the second output point, with this vector being associated with the third output token.

[0028] Furthermore, in some examples, one or more decoders may be configured to determine classifications associated with the output points. As described here, a classification can contain, but is not limited to, a valid point (e.g., a point associated with the characteristic), an invalid point (e.g., a point not associated with the characteristic that can be discarded), a link point (e.g., a point associated with multiple characteristics), and / or any other classification.

[0029] The one or more models can also process the image token and / or output tokens using one or more additional layers configured to generate a heatmap indicating the feature's position as depicted by the image. In some examples, the one or more additional layers can include any type of layer associated with machine learning models, such as one or more convolutional layers. The one or more systems and / or models can then use the output points and / or the heatmap to determine a final representation associated with the feature. Since the output points can be generated using a sequence, for example, the one or more systems and / or models can generate the final representation by combining the points (e.g.,The valid points and / or the connection points are connected according to the order in which the points were generated. Additionally, the one or more systems and / or models can use the heatmap to improve the point connection. By performing such processes, the one or more systems can generate a representation of the feature within the environment, such as a line representation of a road edge, road marking, lane edge, and / or lane marking.

[0030] In some examples, the one or more systems can perform similar processes to generate one or more additional representations for one or more additional features represented by the image and / or one or more additional images. Furthermore, in some examples, the one or more systems can perform one or more processes to determine classifications associated with the features. For example, the one or more systems can process the map, images of the features from the map (e.g., intensity images, color images, etc.), and / or any other information associated with the features to determine the final classifications. For example, if the feature includes a road marking, then the one or more systems can use the images with the one or more models (e.g.,one or more additional layers of the model) and / or one or more additional machine learning models trained to determine whether the road marking includes a solid line, a dashed line, a double line, a center line, a two-way line, an overtaking line, an arrow, a stop line, a pedestrian crossing, and / or any other type of road marking. Additionally, in some examples, the one or more systems may then use these determinations to annotate the map, such as by labeling the map to indicate the locations and / or classifications associated with the features.

[0031] Thus, by performing one or more of the processes described here, the one or more systems are able to determine recent representations associated with features that require no and / or minimal input from one or more users. For example, if the one or more systems use only the one or more learnable tokens as the one or more input tokens for the one or more models, the one or more systems can automatically determine recent representations without requiring input from one or more users. For instance, if a feature contains a road marking in an environment, then the one or more systems can use the one or more learnable tokens and the one or more models to automatically determine a line representation associated with the road marking.This offers numerous improvements over conventional systems, which in turn require users to select hundreds and / or thousands of points to generate such line representations, and / or simply use curve fitting to generate such line representations.

[0032] In some examples, one or more systems can train one or more models to perform one or more of the processes described here, such as determining the information associated with the features. The one or more models can be trained, for example, using training input data, such as data representing training images (e.g., top-down and / or intensity images), depicting the features and / or positions of prompt points associated with the features, and corresponding ground-truth data, such as data representing output points (e.g., actual points) associated with the features, heatmaps indicating the positions of the features, and / or classifications associated with the output points.One or more training engines can then be configured to determine one or more losses using outputs from the one or more models, as generated based on the processing of the training input data, and the ground truth data. For example, the one or more training engines can determine the one or more losses based on at least a comparison of the outputs with the ground truth data using one or more loss functions. The one or more training engines can also be configured to update one or more parameters and / or one or more weights associated with the one or more models using the one or more losses. The training of the one or more models is described in more detail.

[0033] While the examples here describe the processing of images associated with a map to determine information related to features represented by the map, other examples may employ similar processes for other technologies. For instance, a machine navigating within an environment may generate images using one or more sensors, such as one or more red-green-blue images, one or more intensity images, one or more top-down images, and / or any other type of image. The machine may then process the images using one or more of the processes described here to determine information related to features in the environment. Additionally, the machine may perform one or more operations based on at least the information related to the features.If the features include, for example, road markings, the machine can determine how to navigate within the environment based at least on the positions of the road markings in the surroundings.

[0034] In some examples, one or more of the machine learning models described here (e.g., deep neural networks, language models, LLMs, VLMs, multimodal language models, perceptual models, tracking models, fusion models, transformer models, diffusion models, encoder-only models, decoder-only models, encoder-decoder models, neural rendering models (NERF), etc.) can be packaged as a microservice, such as an inference microservice (e.g., NVIDIA's NIMs), which can contain a container (e.g., an operating system (OS)-level virtualization package) that may contain an application programming interface (API) layer, a server layer, a runtime layer, and / or a model engine. For example, the inference microservice can contain the container itself and one or more models (e.g., weights and biases). In some cases, such as...If the one or more machine learning models are small enough (i.e., have a sufficiently small number of parameters), they can be contained within the container itself. In other examples, such as when the one or more models are large, they can be hosted / stored in the cloud (e.g., in a data center) and / or on-premises and / or at the edge (e.g., on a local server or computing device, but outside the container). In such implementations, the one or more models can be accessed via one or more APIs, such as REST APIs.Therefore, in some embodiments, one or more of the machine learning models described herein can be deployed as an inference microservice to accelerate the deployment of one or more models in any cloud, data center, or edge computing system while ensuring data security. For example, the inference microservice may include one or more APIs, a pre-configured container for simplified deployment, an optimized inference engine (e.g., built using standardized software for deploying and running AI models, such as NVIDIA's Triton inference server), and / or one or more APIs for high-performance deep learning inference, which may include inference runtime and model optimizations that provide low latency and high throughput for production applications, such as...The inference microservice may include NVIDIA's TensorRT) and / or enterprise management data for telemetry (e.g., including identity, metrics, health checks, and / or monitoring). The one or more machine learning models described herein may be included as part of the microservice along with accelerated infrastructure capable of being deployed with a single command and / or orchestrated and automatically scaled using a container orchestration system on accelerated infrastructure (e.g., from a single device to the size of a data center). Therefore, the inference microservice may include the one or more machine learning models (e.g., optimized for high-performance inference), inference runtime software to execute the one or more machine learning models, and outputs / responses to inputs (e.g., user queries, prompts, etc.).) and include enterprise management software to provide health checks, identity verification, and / or other monitoring. In some embodiments, the inference microservice may include software to perform an on-premises exchange and / or update of one or more machine learning models. During the exchange or update, the software performing the exchange / update may retain the user configurations of the inference runtime software and the enterprise management software.

[0035] Furthermore, in some embodiments, the systems and procedures described here can be performed within a simulation environment (e.g., NVIDIA's DriveSIM) using simulated data (e.g., simulated sensor data from simulated sensors of a virtual or simulated machine). Simulated sensor data and / or map data (simulated or real) can be used, for example, to perform various operations within the simulation environment, such as determining the positions of features. These simulation operations can be used to test the performance of the underlying algorithms, systems, and / or processes before deployment in the real world. In some cases, simulation can be used to generate synthetic training data, such as training data containing landmarks, features, objects, etc.to contain, generate, so that the synthetic training data (in addition to or as an alternative to real data) can then be processed to perform feature recognition, update virtual or simulated maps and / or perform other operations.

[0036] In each example, such as when a simulation environment is used for testing, validation, training, etc., the simulation environment and / or the associated training data can be rendered or otherwise generated using one or more light transport algorithms, such as ray tracing and / or path tracing algorithms. In some embodiments, the simulation environment and / or one or more objects, features, or components thereof can be generated or managed within a three-dimensional (3D) content collaboration platform (e.g., NVIDIA's OMNIVERSE) for industrial digitization, generative physical AI, and / or other use cases, applications, or services. For example, the content collaboration platform or system can be a system for using or developing Universal Scene Descriptor (USD) data (e.g.,OpenUSD) for managing objects, features, scenes, etc., within a simulation environment, digital environment, etc. The platform can include real-world physics simulation, such as using NVIDIA's PhysX SDK, to simulate real-world physics and physical interactions with simulations hosted by the platform. The platform can integrate OpenUSD, along with ray tracing / path tracing / light transport simulation (e.g., NVIDIA's RTX rendering technologies), into software tools and simulation workflows for building, training, deploying, or testing AI systems, such as systems for testing, validating, and training (e.g., machine learning models, neural networks, etc.) and / or other tasks related to automotive, robotics, machinery, or other applications.

[0037] The systems and procedures described here can be used without restriction by non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g. in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, steered and unsteered robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled with one or more trailers, hydrofoils, boats, shuttles, emergency vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, underwater vehicles, drones and / or other vehicle types.Furthermore, the systems and methods described here can be used for a variety of purposes, including but not limited to machine control, machine locomotion, machine propulsion, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and surveillance, simulation and digital twinning, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital twinning, data center processing, conversational AI, light transport simulations (e.g., ray tracing, path tracing, etc.), collaborative content creation for 3D assets, cloud computing, and / or other suitable applications.

[0038] The disclosed embodiments can include a variety of different systems, such as automotive systems (e.g., a control system for an autonomous or semi-autonomous machine, a perception system for an autonomous or semi-autonomous machine), systems implemented with a robot, aviation systems, media systems, boat systems, intelligent area surveillance systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using an edge device, systems implementing large language models (LLMs), systems implementing one or more multimodal language models, systems using or employing one or more microservices, systems that employ one or more machine learning models in a service or microservice together with a virtualization package (e.g.,a container) at the OS level, systems that include one or more virtual machines (VMs), systems that include one or more virtual machines (VMs), systems that perform operations to generate synthetic data, systems that are at least partially implemented in a data center, systems that perform operations with conversational AI, systems that perform light transport simulations, systems that perform collaborative content creation for 3D assets, systems that perform operations with generative AI, systems that are at least partially implemented using cloud computing resources, and / or other types of systems.

[0039] In relation to Fig. 1A, illustrated Fig. Figure 1A shows an exemplary data flow diagram for a process 100 for feature recognition and / or annotation of a map using one or more machine learning models 102, according to some embodiments of the present disclosure. It should be noted that this and other arrangements described herein are presented only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, arrays, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as single or distributed components, or in conjunction with other components, in any suitable combination and at any suitable location.Various functions described herein, which are performed by entities, can be executed by hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in working memory. In some embodiments, the systems, methods, and processes described herein can be implemented using similar components, features, and / or functionality to that of the exemplary autonomous vehicle 1100 from [reference missing]. Fig. 11A-11D, the exemplary calculating device 1200 from Fig. 12 and / or the exemplary data center 1300 from Fig. 13 will be executed.

[0040] The process 100 can, for example, contain one or more segmentation components 104 that receive map data 106 representing one or more maps. As described here, a map can contain any type of map, such as a navigation map, an SD card, an HD card, and / or any other type of map associated with an environment. In some examples, the map can be associated with different layers corresponding to different types of sensor modalities and / or different types of features. For example, the map can be associated with one or more image layers, one or more LiDAR layers, one or more radar layers, one or more standby state layers, one or more lane marking layers, and / or any other type of layer.Additionally, the map can display information associated with features located within the environment, including but not limited to feature positions, feature orientations, feature classifications, and / or any other information. As described here, the features can include traffic features such as road edges, road markings, carriageway edges, lane markings (e.g., a type of road marking), traffic poles, traffic signs, traffic lights, and / or the like, and / or any other type of feature and / or object that may be located in the environment.

[0041] The process 100 can then involve one or more segmentation components 104 segmenting a map into sections representing specific areas of an environment. As described here, in some examples, the map sections can be represented using images, such as top-down images (e.g., bird's-eye view images) and / or intensity images, depicting different areas of the environment. The LiDAR data (e.g., the LiDAR layer) associated with the map and / or point cloud data associated with the LiDAR data can, for example, represent intensities of points within the environment. The LiDAR sensors used to generate the LiDAR data can, for example, measure the intensities of the points at the time light returns to the LiDAR sensors. The intensities can be represented using numbers, such as…Numbers ranging from 0 to 256 (and / or any other range), where the numbers vary based on the composition (e.g., color, texture, material, etc.) of the surface from which the light is reflected. A low number might indicate low reflectivity, while a high number might indicate high reflectivity. In some examples, the intensities may depend on other factors, such as the angles of incidence, the areas of the points, and / or the like.

[0042] Thus, one or more segmentation components can process the LiDAR data and / or the point cloud data to generate the images (e.g., the top-down, intensity images) that are mapped. Since the intensities of the points can vary based on one or more factors, such as the colors of the surfaces associated with the points (e.g., the colors of the surfaces for which light is reflected), the images can indicate the structure of several features located within the environment. For example, if a feature includes a road marking located on a road within the environment, with the road marking painted using a specific color (e.g., white, yellow, etc.), then an image of the environment can show the points associated with the road marking as a different color compared to the points associated with other features, such as...depicting the road surface itself.

[0043] Fig. Figures 2A-2B illustrate, for example, an instance of segmenting a map to generate images representing features within an environment, according to some embodiments of the present disclosure. As illustrated by the example of Fig. As illustrated in Figure 2A, the one or more segmentation components 104 can initially receive a map 202 of an environment, where the environment contains at least features 204(1)-(8) (also referred to in the singular as "feature 204" or in the plural as "features 204"). While the example of Fig. While the feature 204, including road and / or lane markings, is illustrated in 2A, the map 202 in other examples can represent any other type of feature located in the surroundings. The one or more segmentation components 104 can then segment the map 202 into sections 206(1)-(4) (also referred to in the singular as 'section 206' or in the plural as 'sections 206'). While the example of Fig. 2A illustrates the segmentation of map 202 into four sections 206 that do not overlap; in other examples, one or more segmentation components 104 can segment map 202 into any number of sections and / or one or more sections can overlap with one or more other sections.

[0044] Next; and as illustrated by the example of Fig. As illustrated in Figure 2B, the one or more segmentation components 104 can generate at least one image 208 that is assigned to the first section 206(1) of the map 202. In the example of Fig. 2B The image 208 can contain a top-down intensity image that generates one or more segmentation components 104 using LiDAR data and / or point cloud data associated with the map 202 and / or the environment. For example, the pixels of the image 208 can represent intensities of points within the environment measured by one or more LiDAR sensors when the LiDAR data is generated. Thus, the pixels assigned to features 204 can correspond to points containing the first intensities, while the pixels assigned to other features (e.g., the road surface, the surrounding surface, etc.) can correspond to points containing the second, lower intensities. For this reason, the features 204 appear brighter within the image 208 compared to the surrounding environment.

[0045] While the example of Fig. While Figure 2B illustrates the generation of the top-down intensity image 208, in other examples the one or more segmentation components 104 can generate any other type of image. For example, the one or more segmentation components 104 can generate a red-green-blue (RGB) image that corresponds to the first section 206(1) of the map 202, such as by using image data associated with the map 202 and / or the environment. While the example of Fig. As 2A-2B further illustrates the generation of image 208, which is assigned to the first section 206(1) of map 202, in other examples the one or more segmentation components 104 can generate one or more additional images that are assigned to one or more of the sections 206(2)-(4) of map 202.

[0046] Referring again to the example of Fig. 1A The process 100 can include one or more user devices 108 receiving image data 110, representing the images generated by the one or more segmentation components 104. Based at least on receiving the image data 110, the one or more user devices 108 can present the images to one or more users, who are then able to view the images and provide inputs associated with the features as represented by the images. And as described here, the one or more users can, for example, specify one or more positions of one or more points (referred to as "prompt points" in some examples) located on a given feature, as represented by an image.If the feature includes, for example, a road marking, then one or more users can specify the one or more positions of the one or more points located on the road marking. Based on at least the inputs, the process can include the one or more user devices generating and / or outputting point data representing the points entered by the one or more users.

[0047] Fig. Figure 3 illustrates, for example, an example of determining the positions of the prompt points 302(1)-(3) (also referred to in the singular as "prompt point 302" or in the plural as "prompt points 302"), which are associated with features 204 depicted in Figure 208, according to some embodiments of the present disclosure. As shown, the prompt points 302 include at least the first prompt point 302(1), which is located on the first feature 204(1), the second prompt point 302(2), which is located on the second feature 204(2), and the third prompt point 302(3), which is located on the third feature 204(3). While the example of Fig. While only one prompt point 302 is illustrated for each of the features 204, in other examples any number of prompt points can be specified for each feature 204.

[0048] Referring again to the example of Fig. 1A Process 100 can include applying input data associated with the images, as represented by the image data 110, and the prompt points, as represented by the point data 112, to one or more models 102. Based on at least one or more models 102 processing the input data, Process 100 can include one or more models 102 generating and / or outputting feature data 114, which represents information associated with the features as represented by the images. As described here, in some examples without limitation, the information can include output points associated with the features, heatmaps indicating the positions of the features as represented by the images, and / or representations (e.g., lines) showing the positions of the features within the map.

[0049] For further details illustrated Fig. Figure 1B shows an example of an architecture of one or more models 102 that recognize features, according to some embodiments of the present disclosure. As shown, the one or more models 102 can include one or more image encoders 116 configured to process the image data 110 (e.g., a top-down and / or intensity image) to generate one or more image embeddings 118 associated with the image. As described here, the one or more image encoders 116 can include any type of encoder trained to generate the one or more image embeddings 118 associated with the image, such as a vision transformer. Additionally, the size of the one or more image embeddings 118 can be based on the size of the image.

[0050] The one or more models 102 can further contain one or more prompt encoders 120 configured to process the point data 112 (e.g., the one or more prompt points) to generate one or more input tokens 122 (e.g., input prompt tokens) associated with the one or more prompt points. As described here, the one or more prompt encoders 120 can contain any type of encoder trained to generate the one or more input prompt tokens 122 associated with the one or more prompt points. Additionally, an input prompt token 122 can represent a position encoding associated with a prompt point. Thus, the number of input prompt tokens 122 generated by the one or more prompt encoders 120 can depend on the number of prompt points represented by the point data 112.The one or more prompt encoders 120 can, for example, generate a respective input prompt token 122 for one or more (e.g., each) of the prompt points.

[0051] Additionally, in some examples, at least one of the one or more input tokens 122 may contain a learnable token from the training of the one or more models 102. Thus, the one or more models 102 can append (e.g., link, etc.) the one or more input prompt tokens 122 associated with the one or more prompt points to the one or more learnable input tokens 122 to generate the one or more actual tokens 122, which are then further processed by the one or more models 102. In some examples, a learnable input token 122 can help improve the performance of the one or more models 102 because the learnable input token 122 may not be related to the image (e.g., avoiding alignment with the image) and / or may be able to perform global context aggregation from the image features.

[0052] While this example describes how the one or more input tokens 122 contain both the one or more input prompt tokens 122 and the one or more learnable input tokens 122, in other examples (and as indicated by the dashed lines showing optional data and / or components) the one or more learnable input tokens 122 may contain the one or more learnable input tokens 122 without the one or more input prompt tokens 122. For example, the one or more input tokens 122 may contain a set of learnable tokens that are learned by the one or more models 102 during training. In such examples, the one or more models 102 may then not receive point data 112 and / or contain the one or more prompt encoders 120 configured to generate the one or more input prompt tokens 122.In other words, one or more Model 102s can be configured to perform one or more of the processes described here without receiving any input from the one or more users.

[0053] In such examples, the number of learnable tokens 122 can depend on one or more factors, such as the number of features that can be represented by images. For instance, the number of learnable tokens 122 might be equal to or greater than the maximum number of possible features that can be represented by images. This is because, in some examples, a single learnable token 122 can be used to query a single feature, as represented by an image. Thus, a separate learnable token 122 might be required for each feature.

[0054] The one or more models 102 may further include one or more cross-attention components 124 configured to process the one or more image embeddings 118 and the one or more input tokens 122. As described here, the one or more cross-attention components 124 may include one or more cross-attention layers, such as one or more bi-sided cross-attention layers, one or more token image transformers, and / or any other type of processing component configured to perform one or more of the processes described here.The one or more cross-attention components 124 can, for example, be configured based on at least the processing to compare the text features associated with the one or more prompt points with the image features associated with the image in order to learn relationships between the text features and the image features. Thus, in some examples, the dimensions associated with the one or more image embeddings 118 can be similar to the dimensions associated with the one or more input tokens 122.

[0055] For more details illustrated Fig. 1C Examples of cross-attention layers that can be used by one or more machine learning models according to some embodiments of the present disclosure. As shown, the one or more cross-attention components 124 can contain one or more cross-attention layers 126 and / or one or more cross-attention layers 128. For example, the one or more cross-attention components 124 can contain two of the cross-attention layers 126 (and / or any other number of the cross-attention layers 126) followed by four of the cross-attention layers 128 (and / or any other number of the cross-attention layers 128). As shown, the one or more cross-attention layers 126 can perform pairwise self-attention between query tokens and two-way cross-attention between the set of query tokens and image tokens.Next, the one or more cross-attention layers 128 can independently perform self-attention of a query token and bilateral cross-attention between the query token and a copy of the image tokens. Thus, the one or more cross-attention layers 126 can share all query tokens and the same image features, while the one or more cross-attention layers 128 can contain an updated image token for each query token. In particular, in some examples, each query token of the one or more cross-attention layers 128 can independently perform cross-attention with the image tokens, while all query tokens of the one or more cross-attention layers 126 can perform cross-attention with the image tokens.

[0056] In some examples, the one or more cross-attention layers 126 and / or the one or more cross-attention layers 128 can be associated with one or more of the examples described here. The one or more cross-attention layers 126 and / or the one or more cross-attention layers 128 can be used, for example, when the one or more models 102 are able to automatically perform one or more of the processes described here, such as without using the point data 112 (the one or more models 102 use, for example, only the learnable tokens). In other words, the one or more cross-attention layers 126 and / or the one or more cross-attention layers 128 can help the one or more models 102 generate representations for features without user input.

[0057] Referring again to the example of Fig. 1B can contain one or more models 102 such that the one or more cross-attention components 124 generate and / or output one or more output tokens 130 and / or one or more image tokens 132. As described here, the one or more output tokens 130 can be associated with the position information corresponding to the one or more prompt points represented by the point data 112. Additionally, in some examples, the one or more cross-attention components 124 can be configured to generate and / or output a single output token 130 associated with each feature.Even if the point data 112, for example, represent multiple prompt points for a feature, such that the one or more prompt encoders 120 generate multiple input tokens 122, the one or more cross-attention components 124 may still be configured to generate the single output token 130 that represents the position information associated with the prompt points.

[0058] The one or more models 102 may further contain one or more decoders 134 configured to process the one or more output tokens 130 and / or the one or more image tokens 132 to generate and / or output point data 136 representing one or more points associated with a feature. As described here, in some examples, the point data 136 may represent a specific number of points for which the one or more decoders 134 are configured and / or trained to output, such as 100 points (and / or any other number of points). Additionally, the point data 136 may represent the points using coordinates, such as x-coordinate positions and y-coordinate positions associated with the image and / or map, and / or represent the points using a sequence.Furthermore, in some examples, the point data 136 can represent additional information associated with the points, such as classifications assigned to them. As described here, a classification, without restriction, can contain a valid point (e.g., a point assigned to the characteristic), an invalid point (e.g., a point not assigned to the characteristic, which can be discarded), a junction point (e.g., a point assigned to multiple characteristics), and / or any other type of classification.

[0059] For more details illustrated Fig. 1D is an example of one or more models 102 that output information associated with points of a feature, according to some embodiments of the present disclosure. In the exemplary Fig. In 1D, the one or more decoders 134 can initially process the first output token 130(1), generated by the one or more cross-attention components 124 (and in some examples, the one or more image tokens 132), to generate the first point data 136(1), representing a first point associated with the feature. As shown, the first point can be associated with a first x-coordinate position 138(1), a first y-coordinate position 140(1), and a first classification 142(1). The one or more decoders 134 can then process the second output token 130(2), corresponding to the first point (and in some examples, the one or more image tokens 132 and / or the first output token 130(1)), to generate the second point data 136(2), representing a second point associated with the feature.As shown, the second point can be assigned to a second x-coordinate position 138(2), a second y-coordinate position 140(2) and a second classification 142(2).

[0060] The one or more decoders 134 can then process the third output token 130(3), corresponding to the second point (and in some cases to the one or more image tokens 132 and / or the output tokens 130(1)-(2)), to generate the third point data 136(3), which represents a third point associated with the feature. As shown, the third point can be associated with a third x-coordinate position 138(3), a third y-coordinate position 140(3), and a third classification 142(3). The one or more decoders 134 can then continue these processes until the one or more decoders 134 process the last output token 130 (N) corresponding to the penultimate point (and in some examples the one or more image tokens 132 and / or any other preceding output token) to generate the last point data 136(N) representing a final point associated with the feature.As shown, the last point can be assigned to a last x-coordinate position 138(N), a last y-coordinate position 140(N) and a last classification 142(N).

[0061] As described here, one or more decoders 134 can be configured to generate the point data 136 representing the given number of points, such as 100 points (and / or any other number of points). However, only a portion of the points can actually be assigned to the feature, such as 40 points (and / or any other number of points). Thus, the classifications 142(1)-(N) (also referred to in the singular as "classification 142" or in the plural as "classifications 142") can indicate which of the points are assigned to the feature and which of the points are not. For example, the points assigned to the valid point classification and / or the classification of the connecting point may be assigned to the feature, while the points assigned to the invalid point classification may not be assigned to the feature (the points may, for example, be...be discarded).

[0062] Additionally, as shown, one or more Decoder 134s can output the points in a sequence containing the first point, followed by the second point, followed by the third point, and so on, until the last point. Thus, in some examples, the output points can continue to be assigned to the valid classification and / or the common classification until a point is reached that is assigned to the invalid classification. In this case, the remaining points can then be assigned to the invalid classification, so that these points are not assigned to the characteristic and / or are discarded. As described in more detail here, the sequence in which the points are output can be used to generate a final representation that is assigned to the characteristic.

[0063] While the example of Fig. As illustrated in Figure 1D, the output of a single x-coordinate position 138(1)-(N) and a single y-coordinate position 140(1)-(N) for each point is possible. In other examples, the one or more decoders 134 can output numerous x-coordinate positions and / or numerous y-coordinate positions based on at least one size of the input image. For a point, the one or more decoders 134 can initially determine, for example, a first number of x-coordinate positions based on a size of the input image in the x-coordinate direction, and a second number of y-coordinate positions based on a size of the input image in the y-coordinate direction. For example, if the input image has a size of 1024*512, then the one or more decoders 134 can determine that it has 1024 classes in the x-coordinate direction (e.g., one class per pixel) and 512 classes in the y-coordinate direction (e.g., 1024*512).(one class per pixel). The one or more Decoder 134 (and / or another processing component) can then use the classifications to determine the last position assigned to the point.

[0064] The one or more Decoder 134s can, for example, select a threshold for the results of the highest x-coordinate class and a threshold for the results of the highest y-coordinate class. As described here, the threshold can contain any number of results, such as three results for the x-coordinate classes and three results for the y-coordinate classes. Additionally, the highest coordinate classes can include those classes that are assigned the highest probabilities among the classes. The one or more Decoder 134s can then determine a number of candidate positions using at least the highest x-coordinate classes and the highest y-coordinate classes. For example, if the one or more Decoder 134s determine the three highest x-coordinate classes and the three highest y-coordinate classes, then the one or more Decoder 134s can determine nine candidate positions.Using the candidate positions, the one or more Decoders 134 then determine the last position assigned to the point, including the candidate position assigned to the highest overall probability. The one or more Decoders 134 can then perform similar processes for one or more (e.g., each) of the other points assigned to the feature.

[0065] The points are illustrated as a representation Fig. 4 An example of determining points that are associated with features shown in Figure 208, according to some embodiments of the present disclosure. In the example of Fig. 4. Image 402 can correspond to image 208 by containing an RGB image 402 that is assigned to section 206(1) of map 202. As shown, and for feature 204(1), the one or more decoders 134 can determine a sequence of points 404(1)-(O) containing the first point 404(1), followed by the second point 404(2), followed by the third point 404(3), and / or so on (although not all are labeled for clarity) until the last point 404(O) (e.g., the last valid point and / or connection point). The one or more decoders 134 can then perform similar processes to determine points 406 (although only one is labelled for clarity), which are assigned to the second feature 204(2), and points 408 (although only one is labelled for clarity), which are assigned to the third feature 204(3).

[0066] Referring again to the example of Fig. 1B The one or more models 102 can contain one or more convolutional layers 144 configured to process the one or more output tokens 130 and / or the one or more image tokens 132 to generate and / or output line data 146 representing a line associated with a feature. As described here, in some examples the line data 146 can represent a heatmap indicating the position of the feature as depicted by the image. For example, if the feature includes a road marking, then the heatmap can indicate a line representing the position of the road marking within the image. In other examples, however, the line data 146 can represent any other type of representation associated with the feature, such as a color image and / or an intensity image indicating the position of the feature associated with the input image.Additionally, in other examples, other types of layers of one or more models 102 can be used to generate the line data 146.

[0067] Fig. Figure 5 illustrates, for example, an example of a heatmap 502 that indicates the positions of the features 204(1)-(3), as depicted in Figure 208, according to some embodiments of the present disclosure. As shown, the one or more convolutional layers 144 can generate the heatmap 502, which includes at least a first representation 504(1) indicating the position of the first feature 204(1) within Figure 208, a second representation 504(2) indicating the position of the second feature 204(2) within Figure 208, and a third representation 504(3) indicating the position of the third feature 204(3) within Figure 208. In the example of Fig. 5. The representations may contain lines 504(1)-(3) because features 204(1)-(3) include road markings located within the environment. However, in other examples, representations for other types of features may contain any other shapes that best represent the features as depicted in pictures.

[0068] Referring again to the example of Fig. 1A and as described above for a feature, the feature data 114 can represent the point data 136, which represent the points associated with the feature, as represented by the input image and / or the line data 146, which represent the feature's heatmap. Thus, the process 100 can include one or more processing components 148 (which in some examples may be part of the one or more models 102, such as one or more additional layers) configured to process the feature data 114 to determine a final representation associated with the feature. For example, if the feature contains a road marking, then the one or more processing components 148 can be configured to determine an end line representing the road marking using at least the points associated with the road marking and the heatmap associated with the road marking.

[0069] Since one or more decoders 134 can be configured, for example, to determine the points in the sequence, one or more processing components 148 can be configured to connect at least the points in a sequence assigned to it. Additionally, one or more processing components 148 can be configured to connect specific points, such as points classified as valid points or points classified as connection points. Furthermore, in some examples, one or more processing components 148 can use the heatmap to better connect the points. One or more processing components 148 can then be configured to generate and / or output final feature data 150, representing one or more of the last representations of the one or more features as depicted by the input image.

[0070] In some examples, and as illustrated by the example of Fig. As further illustrated in Figure 1A, the process 100 can include one or more classification components 152 that process at least some of the map data 106 and / or image data 110 to generate classification data 154, which represent the classifications associated with features. As described here, the one or more classification components 152 can include and / or use one or more machine learning models, one or more neural networks, one or more algorithms, one or more classifiers, one or more modules, and / or any other type of processing component to determine the classifications associated with the features. Additionally, a classification associated with a feature can specify at least one type associated with the feature.For example, if a feature includes a road marking, then the classification may include a solid road marking, a dashed road marking, a double road marking, a center line marking, a two-way road marking, an overtaking road marking, an arrow, a stop line, a pedestrian crossing and / or any other type of road marking that may be in the vicinity.

[0071] Although the example of Fig. 1A illustrates the one or more classification components 152 as separate from the one or more models 102. In other examples, the one or more classification components 152 may contain at least a part of the one or more models 102. For example, the one or more classification components 152 may contain one or more layers of the one or more models 102, such that the one or more models 102 are further trained to generate the classification data 154, which represent the classifications associated with the features.

[0072] In some examples, the process 100 may include one or more annotation components 156, which use at least some of the final feature data 150 and / or at least some of the classification data 154, to annotate the map represented by the map data 106. The one or more annotation components 156 may, for example, annotate the map to include at least labels for the locations of the features, the classifications assigned to the features, and / or any other information associated with the features. Thus, by executing the process 100, the map can be automatically annotated with little or no input from one or more users.

[0073] For example, it illustrates Fig. 6 An example of annotating map 202 to include information associated with features 204, according to some embodiments of this disclosure. As shown, the one or more annotation components 156 can use final feature data representing the positions of the features 204 and / or classification data representing the classifications associated with the features 204 to annotate map 202. For example, the one or more annotation components 156 can annotate map 202 to include at least each label 602(1)-(8) (also referred to in the singular as "Label 602" or in the plural as "Labels 602") associated with the features 204. As described here, the Labels 602 can indicate at least the positions of the features 204 and / or the classifications associated with the features 204.

[0074] In some examples, one or more models 102 can be trained to perform one or more of the processes described here. Fig. Figure 7 illustrates, for example, a data flow diagram depicting a process 700 for training one or more models 102 to recognize features associated with images and / or maps, according to some embodiments of the present disclosure. As shown, the one or more models 102 can be trained using training input data 702. In some examples, the training input data 702 can include image data, point data, image embeddings, and / or input tokens. For example, the training input data 702 can be similar to and / or include the image data 110, the point data 112, the image embeddings 118, and / or the input tokens 122.

[0075] The one or more models 102 can be trained using the training input data 702 together with corresponding ground truth data 704. As shown, the ground truth data 704 can represent at least points 706 associated with features, line representations 708 associated with features, and / or classifications 710 associated with the features of the training input data. For example, in some examples, the points 706 can specify the coordinate positions assigned to the points as they are located on the features. Additionally, in some examples, the line representations 708 can include masks and / or heatmaps indicating the positions of the features. Furthermore, in some examples, the classifications 710 can indicate whether the points 706 contain valid points, invalid points, connection points, and / or any other classifications of points.As described here, ground truth data 704 can be synthetically generated (e.g., generated from computer models or renderings), physically generated (e.g., designed and generated from real data), machine-automated (e.g., using feature analysis and learning to extract features from data and then generate labels), annotated by a human (e.g., a labeler or annotation expert defines the location of the labels), and / or a combination thereof. In some examples, there may be corresponding ground truth data 704 for each instance of training input data 702.

[0076] How Fig. As further illustrated in Figure 7, one or more training engines 712 can use one or more loss functions that measure the loss (e.g., error) in outputs 714 compared to the ground truth data 704. As shown, the outputs 714 can also include predicted points 716, predicted line representations 718, and / or predicted classifications 720. Any type of loss function can be used, such as cross-entropy loss, mean squared error, mean absolute error, mean bias error, line segmentation loss, and / or other types of loss functions. In some examples, different outputs 714 can have different loss functions. For example, the predicted points 716 can contain a first loss function, the predicted line representations 718 can contain a second loss function, and / or the predicted classifications 720 can contain a third loss function.In such examples, the loss functions can be combined to form a total loss, and the total loss can be used to train the one or more models 102 (e.g., to update their parameters). In each example, backward computations can be performed to recursively compute gradients of the one or more loss functions with respect to training parameters. In some examples, weights and biases of the one or more models 102 can be used to compute these gradients.

[0077] Fig. Figure 8 illustrates an example of one or more systems 802 that may be configured to perform at least part of the process described herein, according to some embodiments of the present disclosure. As shown, the one or more systems 802 may include at least one or more processors 804, one or more network interfaces 806, and a memory 808. In some examples, the one or more processors 804 may include, or be similar to, one or more CPU(s) 1118, one or more GPU(s) 1120, one or more CPU(s) 1206, and / or one or more GPU(s) 1208. Additionally, the one or more network interfaces 806 may include, or be similar to, one or more network interfaces 1124 and / or one or more communication interfaces 1210. Furthermore, the memory 808 may include, or be similar to, a memory 1204.

[0078] The memory 808 can store one or more models 102, map data 106, one or more segmentation components 104, one or more processing components 148, one or more classification components 152, and / or one or more annotation components 156. Furthermore, the one or more processors 804 can be configured to execute one or more models 102, one or more segmentation components 104, one or more processing components 148, one or more classification components 152, and / or one or more annotation components 156 to perform one or more of the processes described herein.

[0079] As further illustrated by the example of Fig. As illustrated in Figure 8, the one or more systems 802 can communicate with the one or more user devices 108. For example, the one or more systems 802 can send at least the image data 110 to the one or more user devices 108 so that the one or more user devices 108 can present the images to the one or more users. After receiving the user input, the one or more user devices 108 can then send the point data 112 back to the one or more systems 802. In other words, the one or more systems 802 can communicate with the one or more user devices 108 to receive at least the initial prompt points required to perform the process 100. Fig. to determine which are assigned to 1A.

[0080] In some examples, the one or more models 102 can be packaged as a microservice, such as an inference microservice (e.g., NVIDIA's NIMs), which can contain a container (e.g., an operating system virtualization package) that may contain an application programming interface (API) layer, a server layer, a runtime layer, and / or a model "engine." For example, the inference microservice can contain the container itself and the model (e.g., weights and biases). In some cases, such as when the one or more models 102 are small enough (e.g., when they have a sufficiently small number of parameters), the model can be contained within the container itself.In some embodiments, one or more of the models described herein can be used as an inference microservice to accelerate the deployment of models in any cloud, data center, or edge computing system while ensuring data security. For example, the inference microservice may include one or more APIs, a pre-configured container for simplified deployment, an optimized inference engine (e.g., built using standardized software for deploying and running AI models, such as NVIDIA's Triton inference server, and / or one or more APIs for high-performance deep learning inference, which may include inference runtime and model optimizations that provide low latency and high throughput for production applications, such as NVIDIA's TensorRT), and / or enterprise management data for telemetry (e.g.,including identity, metrics, health checks, and / or monitoring). The one or more models described herein can be included as part of the microservice along with an accelerated infrastructure capable of being deployed with a single command and / or orchestrated and automatically scaled using a container orchestration system on an accelerated infrastructure (e.g., on a single device up to the size of a data center). Therefore, the inference microservice can include the one or more machine learning models (e.g., optimized for high-performance inference), inference runtime software to execute the one or more models and provide outputs / responses to inputs (e.g., user queries, prompts, etc.), and enterprise management software to provide health checks, identity, and other monitoring.In some embodiments, the inference microservice may include software to perform an on-site replacement and / or update of one or more Model 102 units. During the replacement or update, the software performing the replacement / update may retain the user configurations of the inference runtime software and the enterprise management software.

[0081] Referring to Fig. 9 and Fig. Each block of Methods 900 and 1000 described herein comprises a computational process that can be performed using any combination of hardware, firmware, and / or software. Various functions can be performed, for example, by a processor executing instructions stored in main memory. Methods 900 and 1000 can also be embodied as computer-usable instructions stored on computer storage media. Methods 900 and 1000 can be provided by a standalone application, a service or hosted service (standalone or in combination with another hosted service), or a plug-in for another product, to name just a few. Furthermore, these Methods 900 and 1000 are illustrated by way of example with respect to Fig. described in 1A-1C. However, these procedures 900 and 1000 can additionally or alternatively be performed by any system or combination of systems, including, but not limited to, the systems described herein.

[0082] Fig. Figure 9 illustrates a flowchart showing a method 900 for recognizing features associated with images, according to some embodiments of the present disclosure. The method in block B902 may include generating one or more embeddings associated with an image representing a feature. The one or more image encoders 116 may, for example, process the image data 110 representing the image to generate the one or more image embeddings 118 associated with the image. As described here, in some examples the image may contain a section of a map that is annotated to contain feature information. However, in other examples the image may contain any other image, such as an image generated using a machine navigating within an environment.

[0083] Procedure 900 in block B904 can include generating one or more input tokens associated with one or more points corresponding to the feature. For example, the one or more input tokens 122 associated with the feature, as shown in the image, can be generated, where the one or more input tokens 122 are associated with the one or more first points. As described here, in some examples, the one or more prompt encoders 120 can generate at least part of the one or more input tokens 122 using the point data 112, which represent the one or more positions of the one or more first points within the image. Additionally, in some examples, at least part of the one or more input tokens 122 can include one or more learnable tokens that are learned during the training of the one or more models 102.In examples where the tokens 122 contain both the one or more input prompt tokens 122 and the one or more learnable tokens 122, the one or more input prompt tokens 122 can be appended to the one or more learnable tokens 122.

[0084] Method 900 in block B906 can include generating, using one or more machine learning models and based on at least one or more embeddings and one or more input tokens, output data representing one or more second points corresponding to the feature. For example, the one or more cross-attention components 124 can initially process the one or more image embeddings 118 and the one or more input tokens 122 to generate at least one or more output tokens 130 and one or more image tokens 132. The one or more decoders 134 can then process the one or more output tokens 130 and / or the one or more image tokens 132 to generate the point data 136 representing the one or more second points associated with the feature.In some examples, the one or more convolution layers 144 can further process the one or more output tokens 130 and / or the one or more image tokens 132 to generate the line data 146 that represent the heatmap of the feature.

[0085] The procedure 900 in block B908 can include determining, based on at least one or more second points, a representation of the feature. The one or more processing components 148 can, for example, process the point data 136 and / or the line data 146 (e.g., the feature data 114) to generate the final feature data 150, which represents the feature. The one or more processing components 148 can, for example, generate the representation by connecting at least one or more second points according to a sequence based on the sequence for which the one or more decoders 134 determined the one or more second points. As described here, in some examples, if the feature includes a road marking, the representation can include a line representation of the road marking.

[0086] The procedure 900 in block B910 can include performing one or more operations based on at least the representation. In some examples, the one or more annotation components 156 can, for instance, use at least the final feature data 150 to annotate the map to provide information associated with the feature (e.g., the feature's location, the feature's classification, etc.). However, in other examples, one or more additional and / or alternative processes can be performed, such as causing a machine to navigate based on at least the feature's representation and / or location.

[0087] Fig. Figure 10 illustrates a flowchart showing another method 1000 for recognizing features associated with images, according to some embodiments of the present disclosure. The method 1000 in block B1002 may include determining one or more first points associated with a feature as represented by an image corresponding to a map. The one or more first points associated with the feature as represented by the image may, for example, be determined. As described here, in some examples, the one or more first points may be determined based on at least one or more user inputs specifying the one or more first points. Additionally or alternatively, in some examples, the one or more first points may be determined automatically using one or more learnable tokens 122.

[0088] Method 1000 in block B1004 can include determining, using one or more machine learning models and based on at least input data associated with one or more first points and the image, one or more second points associated with the feature. The one or more models 102 can, for example, process the input data representing the one or more first points and the image.In some examples, the processing may include generating the one or more input tokens 122 using the one or more first points, generating the one or more image embeddings 118 using the image, generating the one or more output tokens 130 and / or the one or more image tokens 132 using the one or more cross attention components 124, and then generating the point data 136 representing the one or more second points using the one or more decoders 134.

[0089] The procedure 1000 in block B1006 can include generating a representation of the feature based on at least one or more second points. For example, the one or more processing components 148 can process the point data 136 (e.g., the feature data 114) to generate the final feature data 150, which represents the feature. The one or more processing components 148 can, for example, generate the representation by connecting at least one or more second points according to a sequence based on the sequence for which the one or more second points were determined. As described here, in some examples, if the feature includes a road marking, the representation can include a line representation of the road marking.

[0090] Procedure 1000 in block B1008 can include updating the map, based at least on its representation, to contain information associated with the feature. For example, one or more annotation components 156 can update the map to include information associated with the feature. As described here, in some examples, the information can include at least the feature's location, its classification, and / or any other information associated with the feature. EXEMPLARY AUTONOMOUS VEHICLE

[0091] Fig. Figure 11A is an illustration of an exemplary autonomous vehicle 1100, according to some embodiments of the present disclosure. The autonomous vehicle 1100 (here alternatively referred to as "vehicle 1100") may, without limitation, include: a passenger vehicle, such as a car, truck, bus, emergency service vehicle, shuttle, electric or motorized bicycle, motorcycle, fire engine, police vehicle, ambulance, boat, construction vehicle, underwater vehicle, robotic vehicle, drone, aircraft, a vehicle coupled to a trailer (e.g., a semi-trailer truck used for transporting cargo), and / or another type of vehicle (e.g., one that is unmanned and / or carries one or more passengers).Autonomous vehicles are generally described in terms of automation levels defined by the National Highway Traffic Safety Administration (NHTSA), a division of the U.S. Department of Transportation, and the Society of Automotive Engineers (SAE) standard "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles" (Standard No. J3016-201806, published June 15, 2018, Standard No. J3016-201609, published September 30, 2016, and earlier and future versions of this standard). The Vehicle 1100 may exhibit functionality corresponding to one or more of the Levels 3 through 5 of autonomous driving levels.The Vehicle 1100 can exhibit functionality corresponding to one or more of the Levels 1 to 5 of autonomous driving. For example, depending on its configuration, the Vehicle 1100 may be capable of driver assistance (Level 1), partial automation (Level 2), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5). The term "autonomous," as used here, may encompass any and / or all types of autonomy for the Vehicle 1100 or any other machine, such as fully autonomous, highly autonomous, conditionally autonomous, partially autonomous, assistive autonomy, semi-autonomous, primary autonomous, or any other designation.

[0092] The vehicle 1100 can include components such as a chassis, a vehicle body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other vehicle components. The vehicle 1100 can include a propulsion system 1150, such as an internal combustion engine, a hybrid electric power plant, a pure electric motor, and / or another type of propulsion. The propulsion system 1150 can be connected to a drivetrain of the vehicle 1100, which may include a transmission to enable the propulsion of the vehicle 1100. The propulsion system 1150 can be controlled in response to signals received from the throttle valve or accelerator device 1152.

[0093] A steering system 1154, which may include a steering wheel, can be used to steer the vehicle 1100 (e.g., along a desired path or route) when the drive system 1150 is in operation (e.g., when the vehicle is in motion). The steering system 1154 can receive signals from a steering actuator 1156. The steering wheel is optional for full automation (level 5).

[0094] The brake sensor system 1146 can be used to actuate the vehicle brakes in response to receiving signals from the brake actuators 1148 and / or the brake sensors.

[0095] The one or more controllers 1136, the one or more systems-on-chips (SoCs) 1104 ( Fig. 11C) and / or GPUs, can supply signals (e.g., representing instructions) to one or more components and / or systems of the vehicle 1100. For example, the one or more controllers can send signals to actuate the vehicle brakes via one or more brake actuators 1148, to actuate the steering system 1154 via one or more steering actuators 1156, and to actuate the propulsion system 1150 via one or more throttle / accelerator devices 1152. The one or more controllers 1136 can include one or more built-in (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and issue operating commands (e.g., signals representing commands) to enable autonomous driving and / or to assist a human driver in driving the vehicle 1100.The one or more controllers 1136 can include a first controller 1136 for autonomous driving functions, a second controller 1136 for functional safety functions, a third controller 1136 for artificial intelligence functions (e.g., computer vision), a fourth controller 1136 for infotainment functions, a fifth controller 1136 for emergency redundancy, and / or other controllers. In some examples, a single controller 1136 can perform two or more of the above-mentioned functionalities, two or more controllers 1136 can perform a single functionality, and / or any combination thereof.

[0096] The one or more controllers 1136 can provide the signals for controlling one or more components and / or systems of the vehicle 1100 in response to sensor data received from one or more sensors (e.g. sensor inputs). The sensor data can be received, for example, without restriction, from one or more of the following: Global Navigation Satellite Systems (GNSS) sensor(s) 1158 (e.g., Global Positioning System sensor(s)), radar sensor(s) 1160, ultrasonic sensor(s) 1162, lidar sensor(s) 1164, inertial measurement unit (IMU) sensor(s) 1166 (e.g., accelerometer(s), gyroscope(s), magnetic compass(s), magnetometer(s), etc.), microphone(s) 1196, stereo camera(s) 1168, wide-angle camera(s) 1170 (e.g., fisheye cameras), infrared camera(s) 1172, ambient camera(s) 1174 (e.g.,360-degree cameras), long-range and / or medium-range camera(s) 1198, speed sensor(s) 1144 (e.g. for measuring the speed of the vehicle 1100), vibration sensor(s) 1142, steering sensor(s) 1140, brake sensor(s) (e.g. as part of the brake sensor system 1146), and / or other sensor types.

[0097] One or more of the controllers 1136 can receive inputs (e.g., in the form of input data) from an instrument cluster 1132 of the vehicle 1100 and provide outputs (e.g., in the form of output data, display data, etc.) via a human-machine interface (HMI) display 1134, an acoustic alarm, a loudspeaker, and / or via other components of the vehicle 1100. The outputs can include information such as vehicle speed, engine speed, time, map data (e.g., the high-definition (HD) map 1122). Fig. 11C), location data (e.g., the location of vehicle 1100, e.g., on a map), direction, location of other vehicles (e.g., an occupancy grid), information about objects and the status of objects as perceived by the one or more controllers 1136, etc. For example, the HMI display 1134 can show information about the presence of one or more objects (e.g., a road sign, a warning sign, a changing traffic light, etc.) and / or information about driving maneuvers that the vehicle has performed, is currently performing, or will perform (e.g., changing lanes now, taking exit 34B in two miles, etc.).

[0098] The vehicle 1100 also includes a network interface 1124, which can use one or more wireless antennas 1126 and / or modems for communication over one or more networks. The network interface 1124 can be suitable, for example, for communication via Long-Term Evolution (LTE), Wideband Code Division Multiple Access (WCDMA), Universal Mobile Telecommunications System (UMTS), Global System for Mobile Communication (GSM), IMT-CDMA Multi-Carrier (CDMA2000), etc. The one or more wireless antennas 1126 can also enable communication between objects in the environment (e.g., vehicles, mobile devices, etc.) using local area networks such as Bluetooth, Bluetooth Low Energy (LE), Z-Wave, ZigBee, etc.and / or low power wide area networks (LPWANs), such as LoRaWAN, SigFox, etc.

[0099] Fig. 11B is an example of camera locations and fields of view for the exemplary autonomous vehicle 1100. Fig. 11A, according to some embodiments of the present disclosure; The cameras and respective fields of view are an exemplary embodiment and are not intended to be limiting. For example, additional and / or alternative cameras may be included and / or the cameras may be located at different locations on the vehicle 1100.

[0100] The camera types may include, but are not limited to, digital cameras designed for use with the components and / or systems of the 1100 vehicle. The one or more cameras may operate at Automotive Safety Integrity Level (ASIL) B and / or another ASIL. Depending on the configuration, the camera types may be capable of any frame rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc. The cameras may use roller shutters, global shutters, another type of shutter, or a combination thereof.In some examples, the color filter array may include a red-clear-clear-clear (RCCC) color filter array, a red-clear-clear-blue (RCCB) color filter array, a red-blue-green (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor color filter array (RGGB), a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, cameras with clear pixels, such as cameras with an RCCC, RCCB, and / or RBGC color filter array, may be used to increase light sensitivity.

[0101] In some examples, one or more of the cameras can be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For instance, a multi-function monocular camera can be installed to provide features including lane departure warning, traffic sign recognition, and intelligent headlight control. One or more of the cameras (e.g., all cameras) can simultaneously record and provide image data (e.g., video).

[0102] One or more cameras can be mounted in a bracket, such as a specially designed (three-dimensional ("3D") printed bracket, to eliminate stray light and reflections from inside the vehicle (e.g., reflections of the dashboard in the windshield) that could interfere with the camera's image acquisition. Regarding the mounting of exterior mirrors, the mirrors can be individually 3D printed so that the camera mounting plate is shaped to fit the mirror. In some cases, the one or more cameras can be integrated into the exterior mirror. For side cameras, the one or more cameras can also be integrated into the four pillars at each corner of the cabin.

[0103] Cameras with a field of view that includes portions of the environment in front of the vehicle (e.g., forward-facing cameras) can be used for surround view to help identify forward paths and obstacles and, with the aid of one or more controllers and / or control SoCs, to provide information critical for creating an occupancy grid and / or determining preferred vehicle paths. Forward-facing cameras can be used to perform many of the same ADAS functions as LiDAR, including emergency braking, pedestrian detection, and collision avoidance. Forward-facing cameras can also be used for ADAS functions and systems that include lane departure warnings (LDW), autonomous cruise control (ACC), and / or other functions such as traffic sign recognition.

[0104] A variety of cameras can be used in a forward-facing configuration, including, for example, a monocular camera platform containing a complementary metal oxide semiconductor (CMOS) color imager. Another example is the 1170 wide-angle camera, which can be used to capture objects moving into the field of view from the periphery (e.g., pedestrians, crossing vehicles, or bicycles). Although in Fig. While Figure 11B illustrates only one wide-angle camera, the vehicle 1100 can contain any number (including zero) of wide-angle cameras 1170. Furthermore, any number of long-range cameras 1198 (e.g., a pair of long-range stereo cameras) can be used for depth-based object detection, particularly for objects for which a neural network has not yet been trained. The one or more long-range cameras 1198 can also be used for object detection and classification, as well as basic object tracking.

[0105] Any number of stereo cameras 1168 can also be included in a forward-facing configuration. In at least one embodiment, one or more of the stereo cameras 1168 can include an integrated control unit comprising a scalable processing unit that can provide programmable logic (“FPGA”) and a multicore microprocessor with an integrated controller area network (“CAN”) or Ethernet interface on a single chip. Such a unit can be used to create a 3D map of the vehicle's surroundings that includes a distance estimate for all points in the image. Alternatively, one or more stereo cameras 1168 can include a compact stereo vision sensor that may contain two camera lenses (one left and one right) and an image processing chip that measures the distance between the vehicle and the target object and processes the generated information (e.g.,Metadata) can be used to activate the autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 1168 can be used in addition to or as an alternative to those described here.

[0106] Cameras with a field of view that includes sections of the environment to the sides of the vehicle 1100 (e.g., side cameras) can be used for the surround view and provide information that is used to create and update the occupancy grid and to generate side-impact collision warnings. For example, one or more surround cameras 1174 (e.g., four surround cameras 1174, as in Fig. (11B illustrated) are positioned on the vehicle 1100. The one or more surround-view cameras 1174 can include one or more wide-angle cameras 1170, one or more fisheye cameras, one or more 360-degree cameras, and / or the like. For example, four fisheye cameras can be mounted at the front, rear, and sides of the vehicle. In an alternative arrangement, the vehicle can use three surround-view cameras 1174 (e.g., left, right, and rear) and one or more other cameras (e.g., a forward-facing camera) as a fourth surround-view camera.

[0107] Cameras with a field of view that includes sections of the area behind the vehicle 1100 (e.g., reversing cameras) can be used for parking assistance, surround view, rear-impact warnings, and creating and updating the occupancy grid. A variety of cameras can be used, including cameras that are also suitable as one or more forward-facing cameras (e.g., one or more long-range and / or medium-range cameras 1198, one or more stereo cameras 1168, one or more infrared cameras 1172, etc.), as described herein.

[0108] Fig. 11C a block diagram of an exemplary system architecture for the exemplary autonomous vehicle 1100 from Fig. 11A, according to some embodiments of the present disclosure. It should be noted that these and other arrangements described herein are presented only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, arrays, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as single or distributed components, or in conjunction with other components, in any suitable combination and at any suitable location. Various functions described herein, which are performed by entities, may be executed by hardware, firmware, and / or software. Various functions may, for example, be performed by a processor executing instructions stored in a working memory.

[0109] Each of the components, features and systems of the 1100 vehicle in Fig. 11C is illustrated as being connected via bus 1102. Bus 1102 may contain a Controller Area Network (CAN) data interface (here alternatively referred to as a "CAN bus"). A CAN bus can be a network within the vehicle 1100 that serves to support the control of various features and functions of the vehicle 1100, such as the operation of brakes, acceleration, braking, steering, windshield wipers, etc. A CAN bus may be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., a CAN ID). The CAN bus can be read to determine the steering wheel angle, vehicle speed, engine speed (rpm), button positions, and / or other vehicle status indicators. The CAN bus may be ASIL B compliant.

[0110] Although bus 1102 is described here as a CAN bus, this should not be interpreted as a limitation. For example, FlexRay and / or Ethernet can be used in addition to or as an alternative to the CAN bus. Furthermore, while a single line is used to represent bus 1102, this is not intended as a restriction. For instance, there can be any number of buses 1102, which may contain one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using a different protocol. In some examples, two or more buses 1102 can be used to perform different functions and / or for redundancy. For example, a first bus 1102 can be used for collision avoidance functionality, and a second bus 1102 can be used for actuation control.In each example, each bus 1102 can communicate with one of the vehicle's components 1100, and two or more buses 1102 can communicate with the same components. In some examples, each SoC 1104, each controller 1136, and / or each computer within the vehicle can have access to the same input data (e.g., inputs from vehicle 1100 sensors) and be connected to a common bus, such as the CAN bus.

[0111] The vehicle 1100 can contain one or more controllers 1136, as shown here in relation to Fig. 11A. The one or more controllers 1136 can be used for a variety of functions. The one or more controllers 1136 can be coupled with one or more of the various other components and systems of the vehicle 1100 and can be used for controlling the vehicle 1100, for the artificial intelligence of the vehicle 1100, for infotainment for the vehicle 1100 and / or the like.

[0112] The vehicle 1100 can contain one or more systems-on-a-chip (SoC) 1104. The SoC 1104 can contain one or more CPUs 1106, one or more GPUs 1108, one or more processors 1110, one or more caches 1112, one or more accelerators 1114, one or more data storage devices 1116, and / or other components and features not illustrated. The one or more SoCs 1104 can be used to control the vehicle 1100 in a variety of platforms and systems. For example, the one or more SoCs 1104 in a system (e.g., the system of the vehicle 1100) can be combined with an HD card 1122, which is connected via a network interface 1124 to one or more servers (e.g., the one or more servers 1178). Fig. 11D) may receive map refreshes and / or updates.

[0113] The one or more CPUs 1106 can contain a CPU cluster or CPU complex (hereinafter referred to as "CCPLEX"). The one or more CPUs 1106 can contain multiple cores and / or L2 caches. In some embodiments, the one or more CPUs 1106 can, for example, contain eight cores in a coherent multiprocessor configuration. In some embodiments, the one or more CPUs 1106 can contain four dual-core clusters, each cluster having a dedicated L2 cache (e.g., a 2 MB L2 cache). The one or more CPUs 1106 (e.g., the CCPLEX) can be configured to support the concurrent operation of clusters, so that any combination of clusters of the one or more CPUs 1106 can be active at any given time.

[0114] The one or more CPUs 1106 can implement power management functions that include one or more of the following features: individual hardware blocks can be automatically clocked when idle to dynamically save power; each core clock can be controlled when the core is not actively executing instructions due to the execution of WFI / WFE instructions; each core can be independently power-controlled; each core cluster can be independently clock-controlled when all cores are clock-controlled or power-controlled; and / or each core cluster can be independently power-controlled when all cores are power-controlled.The one or more CPUs 1106 can also implement an improved power state management algorithm where permissible power states and expected wake-up times are defined, and the hardware / microcode determines the best power state to input for the core, cluster, and CCPLEX. The processing cores can support simplified sequences for inputting the power state to software, offloading the work to the microcode.

[0115] The one or more GPUs 1108 can include an integrated GPU (referred to herein alternatively as an "iGPU"). The one or more GPUs 1108 can be programmable and can be efficient for parallel workloads. The one or more GPUs 1108 can use an extended Tensor instruction set in some examples. The one or more GPUs 1108 can include one or more streaming microprocessors, each of which can contain an L1 cache (e.g., an L1 cache of at least 96 KB), and two or more of the streaming microprocessors can share an L2 cache (e.g., an L2 cache of 512 KB). In some embodiments, the one or more GPUs 1108 can contain at least eight streaming microprocessors. The one or more GPUs 1108 can use one or more application programming interfaces (APIs) for computation.Furthermore, the one or more GPUs 1108 can use one or more parallel computing platforms and / or programming models (e.g. NVIDIA's CUDA).

[0116] The one or more GPUs 1108 can be power-optimized for best performance in automotive and embedded applications. The one or more GPUs 1108 can be manufactured, for example, on a FinFET field-effect transistor. However, this is not a limitation, and the one or more GPUs 1108 can also be manufactured using other semiconductor manufacturing processes. Each streaming microprocessor can contain an array of mixed-precision processing cores, divided into multiple blocks. For example, and without limitation, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA TENSOR COREs for Deep Learning matrix arithmetic, an L0 instruction cache, a warp scheduler, a dispatch unit and / or a 64 KB register file.Furthermore, streaming microprocessors can include independent parallel integer and floating-point data paths to enable efficient execution of workloads with a mix of computations and addressing operations. Streaming microprocessors can include an independent thread scheduling function to allow fine-grained synchronization and cooperation between parallel threads. Streaming microprocessors can also include a combined L1 data cache and a shared memory unit to improve performance while simplifying programming.

[0117] The one or more GPUs 1108 can include high-bandwidth memory (HBM) and / or a 16 GB HBM2 memory subsystem to provide a peak memory bandwidth of approximately 900 GB / second in some examples. In some examples, synchronous graphics random access memory (SGRAM), such as double-data-rate type five synchronous graphics random access memory (GDDR5), can be used in addition to or as an alternative to HBM memory.

[0118] The one or more GPUs 1108 can incorporate a unified memory technology that includes access counters to enable more accurate migration of memory pages to the processor that accesses them most frequently, thereby improving the efficiency of memory areas shared by processors. In some examples, support for Address Translation Services (ATS) can be used so that the one or more GPUs 1108 can directly access the page tables of the one or more CPUs 1106. In such examples, if the Memory Management Unit (MMU) of the one or more GPUs 1108 fails, an address translation request can be sent to the one or more CPUs 1106.In response, the one or more CPUs 1106 can search their page tables for the virtual-physical mapping for the address and send the translation back to the one or more GPUs 1108. This unified memory technology thus enables a single, unified virtual address space for the memory of both the one or more CPUs 1106 and the one or more GPUs 1108, thereby simplifying the programming of the one or more GPUs 1108 and the porting of applications to the one or more GPUs 1108.

[0119] Additionally, the one or more GPUs 1108 can contain an access counter that tracks the frequency of accesses by the one or more GPUs 1108 to the memory of other processors. The access counter can help ensure that memory pages are moved to the physical memory of the processor that accesses them most frequently.

[0120] The one or more SoCs 1104 can contain any number of caches 1112, including those described here. The one or more caches 1112 can, for example, contain an L3 cache that is available to both the one or more CPUs 1106 and the one or more GPUs 1108 (e.g., one that is connected to both the one or more CPUs 1106 and the one or more GPUs 1108). The one or more caches 1112 can contain a write-back cache that can track the states of the rows, for example, by using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). The L3 cache can be 4 MB or larger, depending on the implementation, although smaller cache sizes can also be used.

[0121] The one or more SoCs 1104 can contain one or more arithmetic logic units (ALUs) that can be used to perform processing related to one of the many tasks or operations of the vehicle 1100—such as DNN processing. Additionally, the one or more SoCs 104 can contain one or more floating-point units (FPUs)—or other mathematical or numerical coprocessors—for performing mathematical operations within the system. For example, the one or more SoCs 104 can contain one or more FPUs integrated as execution units into one or more CPUs 1106 and / or one or more GPUs 1108.

[0122] The one or more SoCs 1104 can contain one or more accelerators 1114 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, the one or more SoCs 1104 can contain a hardware acceleration cluster, which may include optimized hardware accelerators and / or a large amount of on-chip memory. The large on-chip memory (e.g., 4 MB SRAM) can enable the hardware acceleration cluster to accelerate neural networks and other computations. The hardware acceleration cluster can be used in conjunction with the one or more GPUs 1108 and offload some of the tasks from the one or more GPUs 1108 (e.g., to free up more cycles of the one or more GPUs 1108 for other tasks). The one or more accelerators 1114 can, for example, be used for specific workloads (e.g.,Perception, convolutional neural networks (CNNs), etc., are used that are stable enough to be suitable for acceleration. The term "CNN" as used here can include all types of CNNs, including region-based or regional convolutional neural networks (RCNNs) and fast RCNNs (e.g., for object detection).

[0123] The one or more Accelerators 1114 (e.g., the Hardware Acceleration Cluster) can include a Deep Learning Accelerator (DLA). The one or more DLAs can include one or more Tensor Processing Units (TPUs) configured to provide an additional ten trillion operations per second for deep learning applications and inference. The TPUs can be accelerators configured and optimized to perform image processing functions (e.g., for CNNs, RCNNs, etc.). The one or more DLAs can also be optimized for a specific set of neural network types and floating-point operations, as well as for inference. The design of the one or more DLAs can deliver more performance per millimeter than a general-purpose GPU and far surpasses the performance of a CPU.The one or more TPUs can perform multiple functions, including a convolution function for a single instance that supports, for example, INT8, INT16 and FP16 data types for both features and weights, as well as post-processor functions.

[0124] One or more DLAs can quickly and efficiently run neural networks, especially CNNs, on processed or unprocessed data for a variety of functions, including, but not limited to: a CNN for object identification and detection using camera sensor data; a CNN for distance estimation using camera sensor data; a CNN for emergency vehicle detection and identification using microphone data; a CNN for facial recognition and vehicle owner identification using camera sensor data; and / or a CNN for security and / or protection-related events.

[0125] The one or more DLAs can execute any function of the one or more GPUs 1108, and by using an inference accelerator, a developer can, for example, allocate either the one or more DLAs or the one or more GPUs 1108 to each function. For example, the developer can concentrate the processing of CNNs and floating-point operations on the one or more DLAs and leave other functions to the one or more GPUs 1108 and / or other accelerators 1114.

[0126] The one or more Accelerators 1114 (e.g., the Hardware Acceleration Cluster) can contain a Programmable Vision Accelerator (PVA), which can also be referred to here as a Computer Vision Accelerator. The one or more PVAs can be designed and configured to accelerate computer vision algorithms for Advanced Driver Assistance Systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. The one or more PVAs can offer a balance between performance and flexibility. Each PVA can, for example, without limitation, contain any number of Reduced Instruction Set Computer (RISC) cores, Direct Memory Access (DMA) cores, and / or any number of vector processors.

[0127] The RISC cores can interact with image sensors (e.g., the image sensors of one of the cameras described here), image signal processors, and / or the like. Each RISC core can contain any amount of memory. Depending on the implementation, the RISC cores can use any number of protocols. In some examples, the RISC cores can run a real-time operating system (RTOS). The RISC cores can be implemented with one or more integrated circuits, application-specific integrated circuits (ASICs), and / or memory devices. For example, the RISC cores can include an instruction cache and / or tightly coupled RAM.

[0128] The DMA can allow components of the PVA(s) to access the system's memory independently of the single or multiple CPUs. The DMA can support any number of features that serve to optimize the PVA, including, but not limited to, support for multidimensional and / or circular addressing. In some examples, the DMA can support up to six or more dimensions of addressing, which may include block width, block height, block depth, horizontal block stepping, vertical block stepping, and / or depth stepping.

[0129] Vector processors can be programmable processors designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing capabilities. In some examples, the PVA may contain a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, one or more DMA machines (e.g., two DMA machines), and / or other peripheral devices. The vector processing subsystem may operate as the primary processing machine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or working memory (e.g., VMEM).A VPU core can contain a digital signal processor, such as a single instruction, multiple data (SIMD) or a very long instruction word (VLIW). The combination of SIMD and VLIW can increase throughput and speed.

[0130] Each vector processor can contain an instruction cache and can be coupled to dedicated memory. Therefore, in some examples, each vector processor can be configured to operate independently of the others. In other examples, the vector processors contained in a particular PVA can be configured to use data parallelism. For example, in some embodiments, the multiple vector processors contained in a single PVA can execute the same computer vision algorithm, but on different regions of an image. In other examples, the vector processors contained in a particular PVA can simultaneously execute different computer vision algorithms on the same image, or even different algorithms on successive images or sections of an image.Among other things, any number of PVAs can be included in the hardware acceleration cluster, and any number of vector processors can be contained in each of the PVAs. Furthermore, one or more PVAs can include additional memory for error-correcting code (ECC) to increase the overall security of the system.

[0131] The one or more Accelerators 1114 (e.g., the hardware acceleration cluster) can include an on-chip computer vision network and SRAM to provide high-bandwidth, low-latency SRAM for the Accelerators 1114. In some examples, the on-chip memory can include at least 4 MB of SRAM, consisting, for example, and without limitation, of eight field-configurable memory blocks accessible to both the PVA and the DLA. Each pair of memory blocks can include an Advanced Peripheral Bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory can be used. The PVA and the DLA can access the memory via a backbone, enabling high-speed memory access for both the PVA and the DLA.The backbone can include an on-chip computer vision network that connects the PVA and DLA to the main memory (e.g., using the APB).

[0132] The on-chip computer vision network can include an interface that, prior to the transmission of control signals / addresses / data, ensures that both the PVA and the DLA are delivering ready-to-use and valid signals. Such an interface can provide separate phases and channels for the transmission of control signals / addresses / data, as well as burst communication for continuous data transmission. This type of interface can conform to ISO 26262 or IEC 61508 standards, although other standards and protocols can also be used.

[0133] In some examples, one or more SoCs 1104 can include a real-time ray tracing hardware accelerator as described in US patent application No. 16 / 101,232, filed on August 10, 2018. The real-time ray tracing hardware accelerator can be used to quickly and efficiently determine the positions and extents of objects (e.g., within a world model) for generating real-time visualization simulations, for radar signal interpretation, for sound propagation synthesis and / or analysis, for simulating SONAR systems, for general wave propagation simulation, for comparison with lidar data for localization purposes, and / or for other functions and / or purposes. In some embodiments, one or more Tree Traversal Units (TTUs) can be used to perform one or more operations related to ray tracing.

[0134] The single or multiple Accelerators 1114 (e.g., the hardware accelerator cluster) have a wide range of applications for autonomous driving. The PVA can be a programmable vision accelerator used for critical processing steps in ADAS and autonomous vehicles. The PVA's capabilities are well-suited to algorithmic domains requiring predictable processing with low power consumption and low latency. In other words, the PVA is well-suited for semi-dense or dense regular computations, even with small datasets, that demand predictable runtimes with low latency and low power consumption. Therefore, in the context of autonomous vehicle platforms, PVAs are designed to execute classic computer vision algorithms, as they are efficient at object detection and operate with integer mathematics.

[0135] According to one embodiment of the technology, the PVA is used, for example, to perform computer stereovision. In some examples, a semi-global matching-based algorithm can be used, although this is not intended as a limitation. Many applications for Level 3-5 autonomous driving require spontaneous motion estimation or stereo matching (e.g., structure of motion, pedestrian detection, lane detection, etc.). The PVA can perform computer stereovision on input from two monocular cameras.

[0136] In some examples, the PVA can be used to perform dense optical flow processing. This involves processing raw radar data (e.g., using a 4D Fast Fourier Transform) to provide processed radar data. In other examples, the PVA is used for time-of-flight depth processing, for example, by processing raw time-of-flight data to deliver processed time-of-flight data.

[0137] The DLA can be used to power any type of network to improve control and driving safety; this includes, for example, a neural network that outputs a confidence score for each object detection. Such a confidence score can be interpreted as a probability or as providing a relative "weighting" of each detection compared to other detections. This confidence score allows the system to make further decisions about which detections should be considered true positives and not false positives. For example, the system can set a confidence threshold and consider only those detections that exceed the threshold as true positives.In an automatic emergency braking (AEB) system, false positive detections would cause the vehicle to automatically initiate emergency braking, which is obviously undesirable. Therefore, only the safest detections should be considered as triggers for AEB. The DLA can employ a neural network for confidence regression. The neural network can use as input at least a subset of parameters, such as the dimensions of the boundary frame, the ground plane estimate (obtained, for example, from another subsystem), the output of the inertial measurement unit (IMU) sensor 1166 correlated with the vehicle's orientation 1100, distance, and 3D position estimates of the object obtained from the neural network and / or other sensors (e.g., one or more LiDAR sensors 1164 or one or more radar sensors 1160).

[0138] The one or more SoCs 1104 can contain the one or more data stores 1116 (e.g., main memory). The one or more data stores 1116 can be on-chip main memory on the one or more SoCs 1104, where neural networks can be stored to run on the GPU and / or the DLA. In some examples, the one or more data stores 1116 can be large enough to store multiple instances of neural networks for redundancy and security. The one or more data stores 1112 can include one or more L2 or L3 caches 1112. The reference to the one or more data stores 1116 can include a reference to the main memory allocated to the PVA, the DLA, and / or one or more other accelerators 1114, as described here.

[0139] The one or more SoCs 1104 can contain one or more processors 1110 (e.g., embedded processors). The one or more processors 1110 can contain a boot and power management processor, which can be a dedicated processor and subsystem to handle boot power and management functions and the associated security enforcement. The boot and power management processor can be part of the boot sequence of the one or more SoCs 1104 and can provide runtime power management services. The boot and power management processor can provide clock and voltage programming, support for system transitions to a low-power state, management of the thermals and temperature sensors of the one or more SoCs 1104, and / or management of the one or more SoCs 1104 power states.Each temperature sensor can be implemented as a ring oscillator whose output frequency is proportional to the temperature, and the one or more SoCs 1104 can use the ring oscillators to detect the temperatures of the one or more CPUs 1106, the one or more GPUs 1108, and / or the one or more accelerators 1114. If it is determined that the temperatures exceed a threshold, the boot and power management processor can enter a temperature fault routine and put the one or more SoCs 1104 into a lower power state and / or put the vehicle 1100 into a chauffeur-to-safe-stop mode (e.g., bring the vehicle 1100 to a safe stop).

[0140] The one or more 1110 processors can also include a number of embedded processors that can serve as an audio processing engine. The audio processing engine can be an audio subsystem that provides full hardware support for multi-channel audio across multiple interfaces and a wide and flexible range of audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core with a digital signal processor and dedicated RAM.

[0141] The one or more 1110 processors can also include an always-on processor machine, which provides the necessary hardware functions to support low-power sensor management and wake-up from use cases. The always-on processor machine can include a processor core, tightly coupled RAM, supporting peripherals (such as timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0142] The one or more 1110 processors can also include a security cluster machine, which contains a dedicated processor subsystem for the security management of automotive applications. The security cluster machine can include two or more processor cores, tightly coupled RAM, supporting peripherals (such as timers, an interrupt controller, etc.), and / or routing logic. In a security mode, the two or more cores can operate in lockstep mode, functioning as a single core with comparison logic that detects any differences between their operations.

[0143] The one or more 1110 processors can also contain a real-time camera machine, which may include a dedicated processor subsystem for managing the real-time camera.

[0144] The one or more 1110 processors may also include a high dynamic range signal processor, which may include an image signal processor, which is a hardware machine that is part of the camera processing pipeline.

[0145] The one or more 1110 processors can include a video image compositor, which may be a processing block (e.g., implemented on a microprocessor) that implements video post-processing functions required by a video playback application to generate the final image for the player window. The video image compositor can perform lens distortion correction on the one or more 1170 wide-angle cameras, the one or more 1174 ambient lighting cameras, and / or on the sensors of the in-cabin surveillance camera. The in-cabin surveillance camera sensor is preferably monitored by a neural network running on a separate instance of the extended SoC and configured to detect events in the cabin and respond accordingly.A system in the cabin can lip-read to activate mobile service and make a call, dictate emails, change the destination, activate or change the infotainment system and vehicle settings, or enable voice-controlled internet browsing. Certain functions are only available to the driver when the vehicle is operating in autonomous mode and are otherwise deactivated.

[0146] The video image compositor can include enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, if there is motion in a video, the noise reduction weights the spatial information accordingly and reduces the impact of information provided by adjacent frames. If a frame or portion of a frame does not contain motion, the temporal noise reduction performed by the video image compositor can use information from the previous frame to reduce noise in the current frame.

[0147] The video image compositor can also be configured to perform stereo equalization of the input stereo lens images. Furthermore, the video image compositor can be used for user interface design when the operating system desktop is in use and the one or more GPUs 1108 do not need to constantly render new surfaces. Even when the one or more GPUs 1108 are powered on and actively performing 3D rendering, the video image compositor can be used to offload the workload from the GPUs 1108, thus improving performance and responsiveness.

[0148] The one or more 1104 SoCs can also include a serial camera interface with a Mobile Industry Processor Interface (MIPI) for receiving video and camera input, a high-speed interface, and / or a video input block that can be used for camera and related pixel input functions. The one or more 1104 SoCs can also include one or more input / output controllers, one or more of which can be software-controlled and used for receiving I / O signals that are not assigned to a specific role.

[0149] The one or more 1104 SoCs can also include a wide range of peripheral interfaces to enable communication with peripheral devices, audio codecs, power management, and / or other devices. The one or more 1104 SoCs can be used to process data from cameras (e.g., via Gigabit Multimedia Serial Link and Ethernet), sensors (e.g., one or more 1164 LiDAR sensors, one or more 1160 radar sensors, etc., which can be connected via Ethernet), data from the 1102 bus (e.g., vehicle speed 1100, steering wheel position, etc.), and data from one or more 1158 GNSS sensors (e.g., connected via Ethernet or CAN bus).Furthermore, the one or more SoCs 1104 can contain dedicated high-performance mass storage controllers, which can contain their own DMA machines and can be used to offload routine data management tasks from the one or more CPUs 1106.

[0150] The single or multiple 1104 SoCs can form an end-to-end platform with a flexible architecture spanning automation levels 3-5, thereby providing a comprehensive functional safety architecture that supports and efficiently utilizes computer vision and ADAS techniques for diversity and redundancy, and provides a platform for a flexible, reliable driving software stack along with deep learning tools. The single or multiple 1104 SoCs can be faster, more reliable, and even more energy- and space-efficient than conventional systems. For example, the single or multiple 1114 accelerators, in combination with the single or multiple 1106 CPUs, the single or multiple 1108 GPUs, and the single or multiple 1116 data stores, can form a fast, efficient platform for level 3-5 autonomous vehicles.

[0151] This technology thus offers capabilities and functions that cannot be achieved by conventional systems. For example, computer vision algorithms can be run on CPUs that can be configured using a high-level programming language, such as C, to execute a variety of processing algorithms on a wide range of visual data. However, CPUs are often unable to meet the performance requirements of many computer vision applications, such as execution time and power consumption. In particular, many CPUs are unable to execute complex object detection algorithms in real time, which is a requirement for in-vehicle ADAS applications and a prerequisite for practical Level 3-5 autonomous vehicles.

[0152] Unlike conventional systems, the technology described herein, by providing a CPU complex, a GPU complex, and a hardware acceleration cluster, enables the simultaneous and / or sequential execution of multiple neural networks and the combination of their results to enable Level 3-5 autonomous driving functionality. For example, a CNN running on the DLA or the dGPU (e.g., one or more GPUs 1120) can include text and word recognition, allowing the supercomputer to read and understand traffic signs, including signs for which the neural network has not been specifically trained. The DLA can further include a neural network capable of identifying and interpreting the sign, providing a semantic understanding, and passing this semantic understanding to the path planning modules running on the CPU complex.

[0153] Another example is that multiple neural networks can run simultaneously, as required for driving at levels 3, 4, or 5. For instance, a warning sign reading "Caution: Flashing lights indicate black ice" accompanied by an electric light can be interpreted independently or jointly by several neural networks. The sign itself can be identified as a traffic sign by a first neural network (e.g., a trained one), while the text "Flashing lights indicate black ice" can be interpreted by a second neural network, which then informs the vehicle's path planning software (preferably running on the CPU) that the presence of black ice indicates the presence of flashing lights.The turn signal can be identified across multiple images by a third neural network, which informs the vehicle's path planning software about the presence (or absence) of turn signals. All three neural networks can run simultaneously, e.g., within the DLA and / or on one or more GPUs 1108.

[0154] In some examples, a CNN for facial recognition and vehicle owner identification can use data from camera sensors to identify the presence of an authorized driver and / or owner of the vehicle 1100. The always-on sensor processing unit can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and to disable the vehicle in security mode when the owner leaves. In this way, one or more SoCs 1104 provide security against theft and / or carjacking.

[0155] In another example, a CNN for emergency vehicle detection and identification can use data from microphones 1196 to detect and identify emergency vehicle sirens. Unlike conventional systems that use general classifiers to detect sirens and manually extract features, the one or more SoCs 1104 use the CNN to classify environmental and urban sounds as well as visual data. In a preferred embodiment, the CNN running on the DLA is trained to detect the relative approach speed of the emergency vehicle (e.g., by using the Doppler effect). The CNN can also be trained to identify emergency vehicles specific to the local area in which the vehicle is operating, as identified by one or more GNSS sensors 1158.For example, the CNN will attempt to detect European sirens when operating in Europe, and when operating in the United States, the CNN will attempt to identify only North American sirens. Once an emergency vehicle is detected, a controller can be used to execute an emergency vehicle safety routine, slowing the vehicle down, pulling over to the side of the road, parking the vehicle, and / or letting the vehicle idle, using the 1162 ultrasonic sensors, until one or more emergency vehicles pass.

[0156] The vehicle may contain one or more CPUs 1118 (e.g., one or more discrete CPUs or one or more dCPUs) that may be coupled to the one or more SoCs 1104 via a high-speed connection (e.g., PCIe). The CPUs 1118 may, for example, contain an x86 processor. The CPUs 1118 may be used, for example, to perform a variety of functions, including reconciling potentially inconsistent results between ADAS sensors and the one or more SoCs 1104 and / or monitoring the status and health of the one or more Controllers 1136 and / or the Infotainment SoC 1130.

[0157] The Vehicle 1100 can contain one or more GPUs 1120 (e.g., one or more discrete GPUs or one or more dGPUs) that can be coupled to the one or more SoCs 1104 via a high-speed connection (e.g., NVIDIA's NVLINK). The one or more GPUs 1120 can provide additional artificial intelligence capabilities, such as running redundant and / or different neural networks, and can be used to train and / or update neural networks based on inputs (e.g., sensor data) from sensors in the Vehicle 1100.

[0158] The vehicle 1100 may also include the network interface 1124, which may contain one or more wireless antennas 1126 (e.g., one or more wireless antennas for different communication protocols, such as a cellular antenna, a Bluetooth antenna, etc.). The network interface 1124 can be used to establish a wireless connection via the internet to the cloud (e.g., to one or more servers 1178 and / or other network devices), to other vehicles, and / or to computing devices (e.g., client devices of passengers). To communicate with other vehicles, a direct connection between the two vehicles and / or an indirect connection (e.g., via networks and the internet) can be established. Direct connections can be established via vehicle-to-vehicle communication.Vehicle-to-vehicle communication can provide vehicle 1100 with information about vehicles in its vicinity (e.g., vehicles in front of, beside, and / or behind vehicle 1100). This functionality can be part of a cooperative adaptive cruise control function of vehicle 1100.

[0159] The network interface 1124 can include a system-on-a-chip (SoC) that provides modulation and demodulation functions, enabling one or more controllers 1136 to communicate over wireless networks. The network interface 1124 can include a high-frequency (RF) front end for upconversion from baseband to RF and downconversion from RF to baseband. The frequency conversions can be performed using known methods and / or superheterodyne methods. In some examples, the RF front-end functionality can be provided by a separate chip. The network interface can include wireless functionality for communication via LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0160] The vehicle 1100 may further include one or more data storage devices 1128, which may be located outside the chip (e.g., outside the SoCs 1104). The one or more data storage devices 1128 may contain one or more memory elements, including RAM, SRAM, DRAM, VRAM, flash, hard disks, and / or other components and / or devices capable of storing at least one bit of data.

[0161] The 1100 vehicle can also include one or more 1158 GNSS sensors. The one or more 1158 GNSS sensors (e.g., GPS, supported GPS sensors, differential GPS (DGPS) sensors, etc.) assist with mapping, perception, grid generation, and / or path planning. Any number of 1158 GNSS sensors can be used, including, for example, a single GPS unit that uses a USB connection with an Ethernet-to-serial (RS-232) bridge.

[0162] The vehicle 1100 can also include one or more RADAR sensors 1160. The one or more RADAR sensors 1160 can be used by the vehicle 1100 to detect vehicles at long range, even in darkness and / or adverse weather conditions. The functional safety level of the RADAR can be ASIL B. The one or more RADAR sensors 1160 can use the CAN bus and / or the 1102 bus (e.g., for transmitting the data generated by the one or more RADAR sensors 1160) for control and access to object tracking data, with some examples using Ethernet for access to the raw data. A variety of RADAR sensor types can be used. The one or more RADAR sensors 1160 can be suitable for front, rear, and side RADAR applications without restriction. In some examples, one or more pulse-Doppler RADAR sensors are used.

[0163] The single or multiple RADAR 1160 sensors can incorporate various configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range with side coverage, etc. In some examples, long-range RADAR can be used for adaptive cruise control. Long-range RADAR systems can provide a wide field of view, achieved through two or more independent scans, for example, within a range of 250 m. The single or multiple RADAR 1160 sensors can assist in distinguishing between stationary and moving objects and can be used by ADAS systems for emergency braking assistance and frontal collision warning. Long-range RADAR sensors can incorporate a monostatic multimodal RADAR with multiple (e.g., six or more) fixed RADAR antennas and a high-speed CAN and FlexRay interface.In an example with six antennas, the four central antennas can generate a focused beam pattern designed to detect the surroundings of vehicle 1100 at higher speeds with minimal interference from traffic in adjacent lanes. The other two antennas can expand the field of view, enabling the rapid detection of vehicles entering or exiting vehicle 1100's lane.

[0164] Medium-range radar systems, for example, can have a range of up to 1160 m (front) or 80 m (rear) and a field of view of up to 42 degrees (front) or 1150 degrees (rear). Short-range radar systems can include, among other things, radar sensors designed for installation at both ends of the rear bumper. When such a radar sensor system is installed at both ends of the rear bumper, it can generate two beams that continuously monitor the blind spot behind and to the sides of the vehicle.

[0165] Short-range radar systems can be used in an ADAS system for blind spot detection and / or as a lane change assistant.

[0166] The vehicle 1100 can also contain one or more ultrasonic sensors 1162. The one or more ultrasonic sensors 1162, which can be mounted on the front, rear, and / or sides of the vehicle 1100, can be used for parking assistance and / or for creating and updating an occupancy grid. A variety of ultrasonic sensors 1162 can be used, and different ultrasonic sensors 1162 can be used for different detection ranges (e.g., 2.5 m, 4 m). The one or more ultrasonic sensors 1162 can operate with functional safety levels of ASIL B.

[0167] The vehicle 1100 can contain one or more LiDAR sensors 1164. The one or more LiDAR sensors 1164 can be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The one or more LiDAR sensors 1164 can meet the functional safety level ASIL B. In some examples, the vehicle 1100 can contain multiple LiDAR sensors 1164 (e.g., two, four, six, etc.) that can use Ethernet (e.g., to deliver data to a Gigabit Ethernet switch).

[0168] In some examples, one or more LiDAR sensors 1164 may be able to provide a list of objects and their distances for a 360-degree field of view. Commercially available LiDAR sensors 1164 may, for example, have a specified range of approximately 1100 m, with an accuracy of 2 cm to 3 cm and support for an 1100 Mbit / s Ethernet connection. In some examples, one or more non-protruding LiDAR sensors 1164 may be used. In such examples, the one or more LiDAR sensors 1164 may be implemented as a small device that can be embedded in the front, rear, sides, and / or corners of the vehicle 1100. In such examples, one or more LIDAR sensors 1164 can provide a horizontal field of view of up to 120 degrees and a vertical field of view of up to 35 degrees, with a range of 200 m, even with objects of low reflectivity.The one or more front-mounted LIDAR sensors 1164 can be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0169] In some examples, LiDAR technologies, such as 3D flash LiDAR, can also be used. 3D flash LiDAR uses a laser pulse as a transmission source to illuminate the vehicle's surroundings up to approximately 200 m. A flash LiDAR unit contains a sensor that records the travel time of the laser pulse and the reflected light at each pixel, which in turn corresponds to the distance between the vehicle and the objects. Flash LiDAR can enable the generation of highly accurate and distortion-free images of the surroundings with each laser pulse. In some examples, four flash LiDAR sensors can be used, one on each side of the vehicle. Available 3D flash LiDAR systems include a solid-state 3D focal plane array LiDAR camera that contains no moving parts other than a fan (e.g., a non-scanning LiDAR device).The flash LIDAR device can use a 5-nanosecond pulse of a Class I (eye-safe) laser per frame and capture the reflected laser light in the form of 3D distance point clouds and co-registered intensity data. By using flash LIDAR, and because flash LIDAR is a solid-state device with no moving parts, the single or multiple LIDAR sensors can be less susceptible to motion blur, vibration, and / or shock.

[0170] The vehicle may also contain one or more IMU sensors 1166. In some examples, the one or more IMU sensors 1166 may be located in the center of the rear axle of the vehicle 1100. The one or more IMU sensors 1166 may, for example, and without limitation, contain one or more accelerometers, one or more magnetometers, one or more gyroscopes, one or more magnetic compasses, and / or other sensor types. In some examples, such as six-axis applications, the one or more IMU sensors 1166 may contain accelerometers and gyroscopes, while in nine-axis applications, the one or more IMU sensors 1166 may contain accelerometers, gyroscopes, and magnetometers.

[0171] In some embodiments, the one or more IMU sensors 1166 can be implemented as a miniaturized, high-performance GPS-aided inertial navigation system (GPS / INS) that combines inertial sensors of a microelectromechanical system (MEMS), a highly sensitive GPS receiver, and advanced Kalman filter algorithms to provide estimates of position, velocity, and orientation. Thus, in some examples, the one or more IMU sensors 1166 can enable the vehicle 1100 to estimate its course without requiring input from a magnetic sensor by directly observing and correlating velocity changes from the GPS with the one or more IMU sensors 1166.In some examples, one or more IMU sensors 1166 and one or more GNSS sensors 1158 can be combined in a single integrated unit.

[0172] The vehicle may contain one or more microphones 1196, which are mounted in and / or around the vehicle 1100. The one or more microphones 1196 may be used, among other things, for the detection and identification of emergency vehicles.

[0173] The vehicle may also include any number of camera types, including one or more stereo cameras 1168, one or more wide-angle cameras 1170, one or more infrared cameras 1172, one or more surround-view cameras 1174, one or more long-range and / or medium-range cameras 1198, and / or other camera types. The cameras can be used to capture image data around the entire periphery of the vehicle 1100. The types of cameras used depend on the embodiment and requirements of the vehicle 1100, and any combination of camera types can be used to ensure the necessary coverage around the vehicle 1100. Furthermore, the number of cameras can vary depending on the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or any other number of cameras.The cameras can, for example and without limitation, support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet. Each of the one or more cameras is described here with reference to... Fig. 11A and Fig. 11B described in more detail.

[0174] The vehicle 1100 may also contain one or more vibration sensors 1142. The one or more vibration sensors 1142 can measure vibrations of vehicle components, such as one or more axles. For example, changes in vibrations may indicate a change in the road surface. In another example, if two or more vibration sensors 1142 are used, the differences between the vibrations can be used to determine the friction or slippage on the road surface (e.g., if the difference in vibration is between a driven axle and a freely rotating axle).

[0175] The vehicle 1100 may include an ADAS system 1138. In some examples, the ADAS system 1138 may include a SoC. The ADAS system 1138 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward crash warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keep assist (LKA), blind spot warning (BSW), rear cross-traffic warning (RCTW), collision warning systems (CWS), lane centering (LC), and / or other features and functions.

[0176] The ACC systems can use one or more radar sensors, one or more lidar sensors, and / or one or more cameras. The ACC systems can include longitudinal and / or lateral ACC. Longitudinal ACC monitors and controls the distance to the vehicle immediately in front and automatically adjusts the vehicle speed to maintain a safe distance from vehicles ahead. Lateral ACC maintains the distance and advises the vehicle to change lanes if necessary. Lateral ACC interacts with other ADAS applications, such as LCA and CWS.

[0177] The CACC uses information from other vehicles, which can be received via the network interface 1124 and / or the one or more wireless antennas 1126 from other vehicles via a wireless connection or indirectly via a network connection (e.g., via the internet). Direct connections can be provided via a vehicle-to-vehicle (V2V) communication link, while indirect connections can be an infrastructure-to-vehicle (I2V) communication link. Generally, the V2V communication concept provides information about the vehicles immediately ahead (e.g., vehicles directly in front of the vehicle 1100 and in the same lane), while the I2V communication concept provides information about traffic further ahead. CACC systems can incorporate one or both of the I2V and V2V information sources.Given the information about the vehicles ahead of vehicle 1100, the CACC can be more reliable and has the potential to improve traffic flow and reduce congestion on the road.

[0178] FCW systems are designed to warn the driver of a hazard, allowing them to take corrective action. FCW systems use a forward-facing camera and / or one or more RADAR 1160 sensors coupled with a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically connected to the driver feedback system, such as a display, speaker, and / or vibrating component. FCW systems can provide a warning in the form of, for example, an audible signal, a visual warning, a vibration, and / or a rapid braking pulse.

[0179] AEB systems detect an impending head-on collision with another vehicle or object and can automatically apply the brakes if the driver does not take corrective action within a specific time or distance parameter. AEB systems can use one or more forward-facing cameras and / or one or more radar sensors coupled with a dedicated processor, DSP, FPGA, and / or ASIC. When the AEB system detects a hazard, it typically first warns the driver so they can take corrective action to avoid the collision; if the driver does not take corrective action, the AEB system can automatically apply the brakes to prevent or at least mitigate the effects of the predicted collision. AEB systems may incorporate techniques such as dynamic brake assist and / or emergency braking for an impending collision.

[0180] Lane Departure Warning (LDW) systems provide visual, audible, and / or tactile warnings, such as steering wheel or seat vibrations, to alert the driver if the vehicle crosses lane markings. An LDW system will not activate if the driver indicates an intentional lane departure by using a turn signal. LDW systems may utilize forward-facing cameras coupled with a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically connected to the feedback signal for the driver, such as a display, speaker, and / or vibrating component.

[0181] LKA systems are a variant of LDW systems. LKA systems provide steering or braking inputs to correct the vehicle 1100 if the vehicle 1100 begins to leave its lane.

[0182] Blind Spot Warning (BSW) systems detect and warn the driver of vehicles in the car's blind spot. BSW systems can provide a visual, audible, and / or tactile warning to indicate that merging into or changing lanes is unsafe. The system can issue an additional warning if the driver activates a turn signal. BSW systems can use one or more rear-facing cameras and / or radar sensors coupled with a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically connected to the driver for feedback, such as a display, speaker, and / or vibrating component.

[0183] RCTW systems can provide visual, audible, and / or tactile alerts when an object is detected outside the reversing camera's field of view while the vehicle is reversing. Some RCTW systems incorporate AEB (Automatic Emergency Braking) to ensure the vehicle's brakes are applied to prevent a collision. RCTW systems can utilize one or more rear-facing radar sensors coupled with a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically connected to the driver for feedback, such as a display, speaker, and / or vibrating component.

[0184] Conventional ADAS systems can produce false positives, which, while annoying and distracting for the driver, are generally not catastrophic because the ADAS systems warn the driver and give them the opportunity to decide whether a safety issue truly exists and to act accordingly. However, in an autonomous vehicle 1100, the vehicle 1100 itself must decide, in the event of conflicting results, whether to follow the result from a primary computer or a secondary computer (e.g., a first controller 1136 or a second controller 1136). In some embodiments, the ADAS system 1138 can, for example, be a backup and / or secondary computer that provides information about perception to a rationality module of the backup computer.The backup computer rationality monitor can run redundant, diverse software on hardware components to detect errors in perception and dynamic driving tasks. The outputs of the ADAS system 1138 can be provided to a monitoring MCU. If the outputs of the primary and secondary computers conflict, the monitoring MCU must determine how to resolve the conflict to ensure safe operation.

[0185] In some examples, the primary computer can be configured to provide the monitoring MCU with a confidence score indicating its confidence in the chosen outcome. If the confidence score exceeds a threshold, the monitoring MCU can follow the primary computer's instruction, regardless of whether the secondary computer returns a conflicting or inconsistent result. If the confidence score does not reach the threshold and the primary and secondary computers display different results (e.g., conflicting results), the monitoring MCU can mediate between the computers to determine the appropriate outcome.

[0186] The monitoring MCU can be configured to run one or more neural networks trained and configured to determine, based on the outputs of the primary and secondary computers, the conditions under which the secondary computer will trigger false alarms. This allows the one or more neural networks in the monitoring MCU to learn when the secondary computer's output can be trusted and when it cannot. For example, if the secondary computer is a radar-based FCW system, a neural network in the monitoring MCU can learn to trigger an alarm when the FCW system identifies metallic objects that do not actually pose a threat, such as a drain grate or manhole cover.Similarly, if the secondary computer is a camera-based lane departure warning (LDW) system, a neural network in the supervising MCU can learn to override the LDW system when cyclists or pedestrians are present and leaving the lane is indeed the safest maneuver. In embodiments containing one or more neural networks running on the supervising MCU, the supervising MCU can include at least one DLA or GPU suitable for executing the one or more neural networks with associated memory. In preferred embodiments, the supervising MCU can include and / or be contained as a component of the one or more SoCs 1104.

[0187] In other examples, the ADAS system 1138 can include a secondary computer that executes the ADAS functionality according to the classical rules of computer vision. Thus, the secondary computer can use classical computer vision rules (if-then), and the presence of one or more neural networks in the monitoring MCU can improve reliability, safety, and performance. For example, the diverse implementation and intentional non-identity make the overall system more fault-tolerant, especially against errors caused by software (or software-hardware interfaces).For example, if a software bug or error occurs in the software on the primary computer and the non-identical software code on the secondary computer produces the same overall result, the monitoring MCU can have greater confidence that the overall result is correct and that the bug in the software or hardware on the primary computer does not cause a significant error.

[0188] In some examples, the output of the ADAS system 1138 can be fed into the perception block of the primary computer and / or into the dynamic driving task block of the primary computer. For example, if the ADAS system 1138 displays a frontal collision warning due to an object directly in front of the vehicle, the perception block can use this information in object identification. In other examples, the secondary computer may have its own trained neural network, thus reducing the risk of false positives, as described herein.

[0189] The Vehicle 1100 may also include the Infotainment SoC 1130 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system may not actually be an SoC and may contain two or more discrete components. The Infotainment SoC 1130 may include a combination of hardware and software that can be used to provide the Vehicle 1100 with audio (e.g., music, a personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and / or information services (e.g., navigation systems, rear parking sensors, a radio data system, vehicle-related information such as fuel level, total distance traveled, brake fluid level, oil level, door open / close status, air filter information, etc.).The Infotainment SoC 1130 can include, for example, radios, record players, navigation systems, video players, USB and Bluetooth connectivity, car computers, in-car entertainment, Wi-Fi, steering wheel audio controls, hands-free systems, a head-up display (HUD), an HMI display 1134, a telematics device, a control panel (e.g., for controlling and / or interacting with various components, functions, and / or systems), and / or other components. The Infotainment SoC 1130 can also be used to provide information (e.g., visual and / or audible) to one or more vehicle users, such as information from the ADAS system 1138, autonomous driving information such as planned vehicle maneuvers, road layouts, environmental information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0190] The infotainment SoC 1130 can include GPU functionality. The infotainment SoC 1130 can communicate with other devices, systems, and / or components of the vehicle 1100 via the bus 1102 (e.g., CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 1130 can be coupled with a monitoring MCU so that the infotainment system's GPU can perform some self-driving functions if one or more primary controllers 1136 (e.g., the vehicle 1100's primary and / or backup computers) fail. In such an example, the infotainment SoC 1130 can put the vehicle 1100 into a chauffeur-to-safe-stop mode, as described here.

[0191] The vehicle 1100 may also include an instrument cluster 1132 (e.g., a digital instrument panel, an electronic instrument cluster, a digital instrument panel, etc.). The instrument cluster 1132 may include a controller and / or supercomputer (e.g., a discrete controller or supercomputer). The instrument cluster 1132 may contain a number of instruments, such as a speedometer, fuel gauge, oil pressure gauge, tachometer, odometer, turn signals, shift position indicator, seat belt warning light(s), parking brake warning light(s), engine malfunction light(s), airbag system (SRS) information, lighting controls, safety system controls, navigation information, etc. In some examples, information from the infotainment SoC 1130 and the instrument cluster 1132 may be displayed and / or shared. In other words, the Instrument Cluster 1132 can be included as part of the Infotainment SoC 1130, or vice versa.

[0192] Fig. 11D a system diagram for the communication between one or more cloud-based servers and the exemplary autonomous vehicle 1100 from Fig. 11A, according to some embodiments of the present disclosure; The system 1176 may include the one or more servers 1178, the one or more networks 1190, and the vehicles, including the vehicle 1100. The server(s) 1178 may include multiple GPUs 1184(A)-1184(H) (here collectively referred to as GPUs 1184), PCIe switches 1182(A)-1182(H) (here collectively referred to as PCIe switches 1182), and / or CPUs 1180(A)-1180(B) (here collectively referred to as CPUs 1180). The GPUs 1184, the CPUs 1180, and the PCIe switches can be interconnected via high-speed connections, such as, but without limitation, NVIDIA's NVLink interfaces 1188 and / or PCIe connections 1186. In some examples, the GPUs 1184 are connected via NVLink and / or NVSwitch SoCs, and the GPUs 1184 and the PCIe switches 1182 are connected via PCIe connections.Although eight GPUs 1184, two CPUs 1180, and two PCIe switches are illustrated, this should not be interpreted as a limitation. Depending on the configuration, each Server 1178 can contain any number of GPUs 1184, CPUs 1180, and / or PCIe switches. For example, one or more Server 1178s can each contain eight, sixteen, thirty-two, and / or more GPUs 1184.

[0193] The one or more servers 1178 can receive image data from the vehicles via the one or more networks 1190. This image data is representative of images showing unexpected or changed road conditions, such as recently started roadworks. The one or more servers 1178 can transmit neural networks 1192, updated neural networks 1192, and / or map information 1194 to the vehicles via the one or more networks 1190. This map information contains information about traffic and road conditions. The map information updates 1194 can include updates for the HD map 1122, such as information about construction sites, potholes, detours, flooding, and / or other obstacles.In some examples, the neural networks 1192, the updated neural networks 1192 and / or the map information 1194 may result from new training and / or experience represented in the data received from any number of vehicles in the environment, and / or may be based on training performed in a data center (e.g. using one or more servers 1178 and / or other servers).

[0194] One or more Server 1178 systems can be used to train machine learning models (e.g., neural networks) based on training data. The training data can be generated by the vehicles and / or in a simulation (e.g., using a game machine). In some examples, the training data is tagged (e.g., if the neural network benefits from supervised learning) and / or subjected to other preprocessing, while in other examples, the training data is not tagged and / or preprocessed (e.g., if the neural network does not require supervised learning).Training can be performed using one or more classes of machine learning techniques, including, but not limited to, classes such as: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, federated learning, transfer learning, feature learning (including principal component and cluster analysis), multilinear subspace learning, diverse learning, representational learning (including substitute dictionary learning), rule-based machine learning, anomaly detection, and all variants or combinations thereof. Once the machine learning models are trained, they can be used by the vehicles (e.g., transmitted to the vehicles via one or more networks 1190) and / or used by one or more servers 1178 for remote monitoring of the vehicles.

[0195] In some examples, one or more Server 1178 units can receive data from the vehicles and apply that data to advanced neural networks in real time for intelligent, real-time inference. The one or more Server 1178 units can include deep learning supercomputers and / or dedicated AI computers powered by GPUs 1184, such as NVIDIA's DGX and DGX Station machines. However, in some examples, the one or more Server 1178 units can include a deep learning infrastructure that uses only CPU-powered data centers.

[0196] The deep learning infrastructure of one or more servers 1178 can perform fast, real-time inference and can use this capability to assess and verify the state of the processors, software, and / or associated hardware in the vehicle 1100. For example, the deep learning infrastructure can receive periodic updates from the vehicle 1100, such as a sequence of images and / or objects that the vehicle 1100 has located within that sequence of images (e.g., via computer vision and / or other machine learning object classification techniques).The deep learning infrastructure can run its own neural network to identify the objects and compare them with the objects identified by vehicle 1100. If the results do not match and the infrastructure concludes that the AI ​​in vehicle 1100 is not working correctly, one or more servers 1178 can send a signal to vehicle 1100, instructing a fail-safe computer in vehicle 1100 to take control, notify the passengers, and perform a safe parking maneuver.

[0197] For inference, one or more Server 1178 systems can include GPUs 1184 and one or more programmable inference accelerators (e.g., NVIDIA's TensorRT). The combination of GPU-driven servers and inference accelerators can enable real-time responsiveness. In other scenarios, such as when performance is less critical, servers powered by CPUs, FPGAs, and other processors can be used for inference. EXAMPLE CALCULATION DEVICE

[0198] Fig. Figure 12 is a block diagram of an exemplary computing device 1200 suitable for use in implementing some embodiments of the present disclosure. The computing device 1200 may include a connection system 1202 that directly or indirectly couples the following devices: main memory 1204, one or more central processing units (CPUs) 1206, one or more graphics processing units (GPUs) 1208, a communication interface 1210, input / output (I / O) ports 1212, input / output components 1214, a power supply 1216, one or more presentation components 1218 (e.g., display(s)), and one or more logic units 1220. In at least one embodiment, the one or more computing devices 1200 may comprise one or more virtual machines (VMs), and / or each of the components thereof may comprise virtual components (e.g., virtual hardware components).As non-restrictive examples, one or more of the GPUs 1208 can comprise one or more vGPUs, one or more of the CPUs 1206 can comprise one or more vCPUs, and / or one or more of the logic units 1220 can comprise one or more virtual logic units. Thus, a computing device 1200 can contain discrete components (e.g., a complete GPU allocated to the computing device 1200), virtual components (e.g., a portion of a GPU allocated to the computing device 1200), or a combination thereof.

[0199] Although the various blocks of Fig. Where components 12 are shown as connected via the connection system 1202, this is not intended as a limitation and is for clarity only. In some embodiments, for example, a presentation component 1218, such as a display device, can be considered an I / O component 1214 (e.g., if the display is a touchscreen). As another example, the CPUs 1206 and / or GPUs 1208 can contain memory (e.g., the memory 1204 can represent a storage device in addition to the memory of the GPUs 1208, the CPUs 1206, and / or other components). In other words, the computing device of Fig. Section 12 is for illustrative purposes only. No distinction is made between categories such as "workstation", "server", "laptop", "desktop", "tablet", "client device", "mobile device", "handheld device", "game console", "electronic control unit (ECU)", "virtual reality system" and / or other device or system types, as all are within the scope of protection of the computing device of Fig. 12 are being considered.

[0200] The 1202 interconnection system can represent one or more connections or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The 1202 interconnection system can include one or more bus or connection types, such as an Industry Standard Architecture (ISA) bus, an Extended ISA bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI Express (PCIe) bus, and / or another type of bus or connection. In some embodiments, there are direct connections between components. For example, the 1206 CPU can be directly connected to the 1204 main memory. Furthermore, the 1206 CPU can be directly connected to the 1208 GPU.In a direct or point-to-point connection between components, the 1202 connection system can include a PCIe link to establish the connection. In these examples, a PCI bus does not need to be included in the 1200 computing device.

[0201] The 1204 main memory can contain a variety of computer-readable media. Computer-readable media can be any available media that the 1200 computing device can access. Computer-readable media can include both volatile and non-volatile media, as well as removable and non-removable media. For example, and without limitation, computer-readable media can include computer storage media and communication media.

[0202] Computer storage media can include both volatile and non-volatile media, and / or removable and non-removable media, implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, and / or other data types. For example, main memory can store computer-readable instructions (e.g., representing one or more programs and / or one or more program elements, such as an operating system). Computer storage media can include, but are not limited to, RAM, ROM, EEPROM, flash memory, or other storage technologies; CD-ROM, Digital Versatile Discs (DVDs), or other optical disk storage; magnetic cartridges, magnetic tapes, magnetic disk storage, or other magnetic storage devices; or any other medium that can be used to store the desired information and that the computing device can access.As used here, computer storage media do not inherently contain signals.

[0203] Computer storage media can embody computer-readable instructions, data structures, program modules, and / or other data types in a modulated data signal, such as a carrier wave or other transport mechanism, and include any media for transmitting information. The term "modulated data signal" can refer to a signal in which one or more of its properties are set or modified to encode information within the signal. Computer storage media can include, but are not limited to, wired media, such as a wired network or a direct-wired connection, and wireless media, such as acoustic, RF, infrared, and other wireless media. Combinations of the above should also be included in the scope of protection of the computer-readable media.

[0204] The one or more CPUs 1206 can be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1200 to perform one or more of the procedures and / or processes described herein. The one or more CPUs 1206 can each contain one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of processing a multitude of software threads simultaneously. The one or more CPUs 1206 can contain any type of processor and can contain different types of processors depending on the type of computing device 1200 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers).Depending on the type of computing device 1200, the processor can be, for example, an Advanced RISC Machine (ARM) processor implemented with Reduced Instruction Set Computing (RISC), or an x86 processor implemented with Complex Instruction Set Computing (CISC). The computing device 1200 can contain one or more CPUs 1206, in addition to one or more microprocessors or additional coprocessors, such as mathematical coprocessors.

[0205] In addition to or as an alternative to the one or more CPUs 1206, the one or more GPUs 1208 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1200 to perform one or more of the procedures and / or processes described herein. One or more of the GPUs 1208 can be an integrated GPU (e.g., with one or more of the CPUs 1206) and / or one or more of the GPUs 1208 can be a discrete GPU. In embodiments, one or more of the GPUs 1208 can be a coprocessor of one or more of the CPUs 1206. The one or more GPUs 1208 can be used by the computing device 1200 to render graphics (e.g., 3D graphics) or to perform general-purpose computing. The one or more GPUs 1208 can be used, for example, for general-purpose computing on GPUs (GPGPU).The one or more GPUs 1208 can contain hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. The one or more GPUs 1208 can generate pixel data for output images in response to rendering commands (e.g., rendering commands from the one or more CPUs 1206 received via a host interface). The one or more GPUs 1208 can include graphics memory, such as display memory, for storing pixel data or other suitable data, such as GPGPU data. The display memory can be included as part of the 1204 main memory. The one or more GPUs 1208 can contain two or more GPUs operating in parallel (e.g., via a link). The link can connect the GPUs directly (e.g., using NVLINK) or connect them via a switch (e.g., using NVSwitch).When combined, each GPU can generate 1208 pixel data or GPGPU data for different sections of an output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU can have its own dedicated memory or share memory with other GPUs.

[0206] In addition to or as an alternative to the one or more CPUs 1206 and / or the one or more GPUs 1208, the one or more logic units 1220 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1200 to perform one or more of the methods and / or processes described herein. In embodiments, the one or more CPUs 1206, the one or more GPUs 1208, and / or the one or more logic units 1220 may discretely or jointly execute any combination of the methods, processes, and / or sections thereof. One or more of the logic units 1220 may be part of and / or integrated into one or more of the CPUs 1206 and / or one or more of the GPUs 1208, and / or one or more of the logic units 1220 may be discrete components or otherwise separate from the CPUs 1206 and / or the GPUs 1208.In embodiments, one or more of the logic units 1220 can be a co-processor of one or more of the CPUs 1206 and / or one or more of the GPUs 1208.

[0207] Examples of one or more logic units 1220 contain one or more processing cores and / or components thereof, such as data processing units (DPUs), tensor cores (TCs), tensor processing units (TPUs), pixel visual cores (PVCs), vision processing units (VPUs), graphics processing clusters (GPCs), texture processing clusters (TPCs), streaming multiprocessors (SMs), tree traversal units (TTUs), artificial intelligence accelerators (AIAs), deep learning accelerators (DLAs), arithmetic logic units (ALUs), and application-specific integrated circuits. (Application-Specific Integrated Circuits, ASICs), Floating Point Units (FPUs),Input / output (I / O) elements, peripheral component interconnect (PCI) or PCI Express (PCIe) elements, and / or similar.

[0208] The 1210 communication interface can include one or more receivers, transmitters, and / or transceivers that enable the 1200 computing device to communicate with other computing devices over an electronic network, including wired and / or wireless communication. The 1210 communication interface can include components and functions that enable communication over a variety of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., Ethernet or InfiniBand communication), low-power wide-area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet.In one or more embodiments, the one or more logic units 1220 and / or the communication interface 1210 may contain one or more data processing units (DPUs) to transfer data received via a network and / or via the connection system 1202 directly to one or more GPUs 1208 (e.g., a memory thereof).

[0209] The I / O ports 1212 enable the computing device 1200 to be logically coupled with other devices, including the I / O components 1214, one or more presentation components 1218, and / or other components, some of which may be built into (e.g., integrated with) the computing device 1200. Illustrative I / O components 1214 include a microphone, mouse, keyboard, joystick, gamepad, game controller, satellite dish, scanner, printer, wireless device, etc. The I / O components 1214 can provide a natural user interface (NUI) that processes air gestures, speech, or other physiological inputs generated by a user. In some cases, the inputs can be transmitted to a suitable network element for further processing.A NUI can implement any combination of speech capture, stylus capture, face capture, biometric capture, gesture capture (both on-screen and off-screen), air gestures, head and eye tracking, and touch capture (as further described below) associated with a display of the Computing Device 1200. The Computing Device 1200 can include depth cameras, such as stereoscopic camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof, for gesture capture and recognition. Additionally, the Computing Device 1200 can include accelerometers or gyroscopes (e.g., as part of an inertial measurement unit (IMU)) that enable motion detection. In some examples, the output from the accelerometers or gyroscopes can be used by the Computing Device 1200 to render immersive augmented reality or virtual reality.

[0210] The power supply 1216 can include a hardwired power supply, a battery power supply, or a combination thereof. The power supply 1216 can power the computing device 1200 to enable the operation of the computing device 1200's components.

[0211] The one or more presentation components 1218 can include a display (e.g., a monitor, a touchscreen, a television screen, a head-up display (HUD), other display types, or a combination thereof), speakers, and / or other presentation components. The one or more presentation components 1218 can receive data from other components (e.g., the one or more GPUs 1208, the one or more CPUs 1206, DPUs, etc.) and output the data (e.g., as an image, video, sound, etc.). EXEMPLARY DATA CENTER

[0212] Fig. Figure 13 illustrates an exemplary data center 1300 that can be used in at least one embodiment of the present disclosure. The data center 1300 can include an infrastructure layer 1310, a framework layer 1320, a software layer 1330, and / or an application layer 1340.

[0213] As in Fig. As shown in Figure 13, the infrastructure layer 1310 of the data center can contain a resource orchestrator 1312, grouped compute resources 1314 and node compute resources (“node RRs”) 1316(1)-1316(N), where “N” is any positive integer. In at least one embodiment, the node RRs 1316(1)-1316(N) may include, but are not limited to, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field programmable gate arrays (FPGAs), graphics processing units or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic solid-state memory), storage devices (e.g., solid-state or disk drives), network input / output (NW I / O) devices, network switches, virtual machines (VMs), power supply modules and / or cooling modules, etc.In some embodiments, one or more node RRs among node RRs 1316(1)-1316(N) may correspond to a server that has one or more of the compute resources mentioned above. Furthermore, in some embodiments, node RRs 1316(1)-1316(N) may contain one or more virtual components, such as vGPUs, vCPUs, and / or the like, and / or one or more of node RRs 1316(1)-1316(N) may correspond to a virtual machine (VM).

[0214] In at least one embodiment, the grouped compute resources 1314 can contain separate groupings of node RRs 1316, which are housed in one or more racks (not shown) or in many racks in data centers at different geographic locations (also not shown). Separate groupings of node RRs 1316 within grouped compute resources 1314 can contain grouped compute, network, memory, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, multiple node RRs 1316, including the CPUs, GPUs, DPUs, and / or other processors, can be grouped in one or more racks to provide compute resources to support one or more workloads.The one or more racks can also contain any number of power supply modules, cooling modules and / or network switches in any combination.

[0215] The resource orchestrator 1312 can configure or otherwise control one or more node RRs 1316(1)-1316(N) and / or grouped compute resources 1314. In at least one embodiment, the resource orchestrator 1312 can include an entity for managing the software design infrastructure (SDI) for the data center 1300. The resource orchestrator 1312 can include hardware, software, or a combination thereof.

[0216] In at least one embodiment, as in Fig. As shown in Figure 13, the framework layer 1320 can contain a job scheduler 1333, a configuration manager 1334, a resource manager 1336, and / or a distributed file system 1338. The framework layer 1320 can contain a framework that supports the software 1332 of the software layer 1330 and / or one or more applications 1342 of the application layer 1340. The software 1332 or the one or more applications 1342 can each contain web-based service software or applications such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. The framework layer 1320 can be a type of free and open-source software web application framework, such as Apache Spark™ (hereinafter "Spark"), which can use a distributed file system 1338 for processing large amounts of data (e.g., "Big Data"), without being limited to it.In at least one embodiment, the job scheduler 1333 can include a Spark driver to facilitate the scheduling of workloads supported by different layers of the data center 1300. The configuration manager 1334 can be able to configure different layers, such as the software layer 1330 and the framework layer 1320, which contains Spark and the distributed file system 1338, to support the processing of large amounts of data. The resource manager 1336 can be able to manage clustered or grouped compute resources allocated or assigned to support the distributed file system 1338 and the job scheduler 1333. In at least one embodiment, the clustered or grouped compute resources can include the grouped compute resource 1314 on the infrastructure layer 1310 of the data center.The resource manager 1336 can coordinate with the resource orchestrator 1312 to manage these allocated or assigned computing resources.

[0217] In at least one embodiment, the software 1330 contained in software layer 1332 may include software used by at least sections of the node RRs 1316(1)-1316(N), the grouped compute resources 1314, and / or the distributed file system 1338 of framework layer 1320. One or more types of software may include, among others, web page search software, email virus scanning software, database software, and streaming video content software.

[0218] In at least one embodiment, the applications 1342 contained in the application layer 1340 may include one or more types of applications used by at least sections of the node RRs 1316(1)-1316(N), the grouped compute resources 1314, and / or the distributed file system 1338 of the framework layer 1320. One or more types of applications may include, but are not limited to, any number of genome applications, cognitive computations, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments.

[0219] In at least one embodiment, the configuration manager 1334, the resource manager 1336, and / or the resource orchestrator 1312 can implement any number and type of self-modifying actions based on any set and type of data acquired in any technically feasible way. Self-modifying actions can relieve a data center operator of the data center 1300 of potentially making poor configuration decisions and potentially avoiding underutilized and / or poorly functioning sections of a data center.

[0220] The Data Center 1300 may contain tools, services, software, or other resources to train one or more machine learning models or to predict or infer information using one or more machine learning models according to one or more embodiments described herein. For example, one or more machine learning models may be trained by calculating weighting parameters according to a neural network architecture, using software and / or computing resources described above in relation to the Data Center 1300.In at least one embodiment, trained or deployed machine learning models corresponding to one or more neural networks can be used to infer or predict information using the resources described above with reference to the Computing Center 1300 by using weighting parameters calculated by one or more training techniques such as, but not limited to, those described herein.

[0221] In at least one embodiment, the data center can use 1300 CPUs, application-specific integrated circuits (ASICs), GPUs, FPGAs, and / or other hardware (or equivalent virtual computing resources) to perform training and / or inference using the resources described above. Furthermore, one or more of the software and / or hardware resources described above can be configured as a service to allow users to train or infer information, such as image capture, speech capture, or other artificial intelligence services.

[0222] The Data Center 1300 can contain one or more components, such as one or more Central Processing Units (CPUs), one or more Graphics Processing Units (GPUs), one or more Isolated Trusted Execution Environments (TEEs), one or more connections for multi-GPU communication, one or more Data Processing Units (DPUs), and one or more Network Interface Chips (NICs). EXEMPLARY NETWORK ENVIRONMENTS

[0223] Network environments suitable for implementing embodiments of the disclosure may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. The client devices, servers, and / or other device types (e.g., each device) may run on one or more instances of the one or more computing devices. Fig. 12. Implemented - for example, each device may contain similar components, features, and / or functionality to one or more computing devices 1200. If backend devices (e.g., servers, NAS, etc.) are implemented, the backend devices may also be included as part of a data center 1300, an example of which is given herein with reference to Fig. 13 is described in more detail.

[0224] The components of a network environment can communicate with each other over one or more networks, which can be wired, wireless, or both. The network can contain multiple networks or a network of networks. For example, the network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks such as the internet and / or a public switched telephone network (PSTN), and / or one or more private networks. If the network includes a wireless telecommunications network, components such as a base station, a communications tower, or even access points (as well as other components) can provide wireless connectivity.

[0225] Compatible network environments can contain one or more peer-to-peer network environments—in which case a server cannot be included in a network environment—and one or more client-server network environments—in which case one or more servers can be included in a network environment. In peer-to-peer network environments, the functionality described here can be implemented on any number of client devices with reference to one or more servers.

[0226] In at least one embodiment, a network environment can include one or more cloud-based network environments, a distributed computing environment, a combination thereof, etc. A cloud-based network environment can include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. A framework layer can include a framework to support software of a software layer and / or one or more applications of an application layer. The software or the one or more applications may each include web-based service software or applications. In embodiments, one or more of the client devices can use the web-based service software or applications (e.g.,by accessing the service software and / or applications via one or more application programming interfaces (APIs). The framework layer can be a type of free and open-source software web application framework that uses, for example, a distributed file system for processing large amounts of data (e.g., "Big Data"), but is not limited to this.

[0227] A cloud-based network environment can provide cloud computing and / or cloud storage, performing any combination (or one or more parts) of the computing and / or data storage functions described herein. Each of these different functions can be distributed across multiple locations of central or core servers (e.g., one or more data centers, which may be distributed across a state, region, country, the globe, etc.). If a connection to a user (e.g., a client device) is relatively close to one or more edge servers, one or more core servers can offload at least some functionality to the one or more edge servers. A cloud-based network environment can be private (e.g., restricted to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0228] The one or more client devices can incorporate at least some of the components, features, and functions of the one or more devices referred to here. Fig.The 12 described exemplary computing devices may include 1200. By way of example, and not as a limitation, a client device may be a personal computer (PC), a laptop, a mobile device, a smartphone, a tablet computer, a smartwatch, a portable computer, a personal digital assistant (PDA), an MP3 player, a virtual reality headset, a global positioning system (GPS) or global positioning device, a video player, a video camera, a surveillance device or surveillance system, a vehicle, a boat, a hydrofoil, a virtual machine, a drone, a robot, a handheld communication device, a hospital device, a gaming device or gaming system, an entertainment system, a vehicle computer system, an embedded system controller, a remote control, a device, a consumer electronics device, a workstation, an edge device,any combination of these described devices or any other suitable device may be embodied.

[0229] The disclosure can be described in the general context of computer code or machine-usable instructions, including computer-executable instructions such as program modules that are executed by a computer or other machine, such as a personal data assistant or other handheld device. Generally, program modules, which contain routines, programs, objects, components, data structures, etc., refer to code that performs specific tasks or implements certain abstract data types. The disclosure can be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc.The revelation can also be practiced in distributed computing environments, where tasks are performed by remote processing devices that are connected to each other via a network for communication.

[0230] As used herein, any mention of "and / or" in relation to two or more elements should be interpreted as referring to only one element or combination of elements. For example, "element A, element B and / or element C" may include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. Furthermore, "at least one of element A or element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Additionally, "at least one of element A and element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0231] The subject matter of this disclosure is specifically described herein to satisfy legal requirements. However, the description itself is not intended to limit the scope of protection afforded by this disclosure. Rather, the inventors have considered that the claimed subject matter may also be embodied in other ways to include various steps or combinations of steps similar to those described in this document, in conjunction with other present or future technologies. Although the terms “step” and / or “block” may be used herein to denote various elements of the methods employed, these terms should not be interpreted as implying any particular sequence among or between the various steps disclosed herein, except where the sequence of each step is expressly described. EXAMPLE CLAUSES

[0232] A: Method comprising: generating one or more input tokens representing one or more first points associated with a road marking, as depicted by an image associated with a map; generating one or more embeddings associated with the image; generating one or more machine learning models and, based on at least the one or more input tokens and the one or more embeddings, one or more output tokens representing one or more second points associated with the road marking; generating a line representation of the road marking based on at least the one or more second points; and updating the map, based at least on the line representation, to include a label associated with the road marking.

[0233] B: Method according to paragraph A, further comprising at least one of the following: receiving input data representing the one or more first points associated with the road marking; or determining, based on at least analyzing at least one of the map or image, the one or more first points associated with the road marking.

[0234] C: Method according to either paragraph A or paragraph B, wherein generating the one or more output tokens comprises: generating, using the one or more machine learning models and based on at least the one or more input tokens and the one or more embeddings, one or more first output tokens representing a first section of the one or more second points; and generating, using the one or more machine learning models and based on at least the one or more first output tokens, one or more second output tokens representing a second section of the one or more second points.

[0235] D: Method according to any of paragraphs AC, further comprising: generating, using the one or more machine learning models and based on at least the one or more input tokens and the one or more embeddings, one or more image tokens associated with the image, wherein the generation of the line representation is further based on at least the one or more image tokens.

[0236] E: Method according to any of paragraphs AD, further comprising: appending the one or more input tokens to one or more learnable tokens to generate one or more appended input tokens, wherein the generation of the one or more input tokens is further based on at least the one or more appended input tokens and the one or more embeddings.

[0237] F: Method according to any of paragraphs AE, further comprising: Determining, based on at least one or more output tokens, one or more classifications associated with one or more second points, wherein generating the line representation is further based on at least one or more classifications.

[0238] G: Method according to any of paragraphs AF, further comprising: generating, using one or more decoders and based on at least one or more output tokens, one or more coordinates associated with one or more second points within the image, wherein the generation of the line representation is based on at least one or more coordinates.

[0239] H: Method according to any of paragraphs AG, further comprising: generating, based on at least one or more output tokens or one or more image tokens associated with the image, a heatmap associated with the road marking, wherein the generation of the line representation is further based at least on the heatmap.

[0240] I: Data center comprising: one or more central processing units (CPUs); one or more graphics processing units (GPUs); one or more isolated trusted execution environments (TEEs); one or more links for multi-GPU communication; one or more data processing units (DPUs); one or more network interface chips (NICs); wherein one or more components of the data center are used to: determine one or more first points associated with a traffic feature from a sensor data representation corresponding to a map; determine, using one or more machine learning models and based on at least input data associated with the one or more first points and the sensor data representation, the one or more second points associated with the traffic feature;Generating a representation of the traffic feature based on at least one or more second points; and updating the map, based at least on the representation, to include information associated with the traffic feature.

[0241] J: Data center according to paragraph I, wherein the one or more components further serve to: generate one or more input tokens based on at least the one or more first points and one or more embeddings based on at least the sensor data representation, wherein the input data are associated with the one or more input tokens and the one or more embeddings.

[0242] K: Data center according to paragraph J, wherein the one or more components further serve to: append the one or more input tokens to one or more learnable tokens in order to generate one or more appended input tokens, wherein the input data is associated with the one or more appended input tokens and the one or more embeddings.

[0243] L: Data center according to one of paragraphs IK, wherein the determination of the one or more second points associated with the traffic feature comprises: generating, using the one or more machine learning models and based on at least the input data, one or more output tokens; and determining, based on at least the one or more output tokens, the one or more second points associated with the traffic feature.

[0244] M: Data center according to any of paragraphs IL, wherein the one or more components further serve to perform at least one of the following: receiving one or more inputs representing the one or more first points associated with the traffic feature; or determining, based on at least the analysis of at least one of the map or sensor data representations, the one or more first points associated with the traffic feature.

[0245] N: Data center according to one of paragraphs IM, wherein determining the one or more second points associated with the traffic feature comprises: determining, using the one or more machine learning models and based on at least the input data, at least one first section of the one or more second points; and determining, using the one or more machine learning models and based on at least second input data associated with the at least first section of the one or more second points, at least one second section of the one or more second points.

[0246] O: Data center according to paragraph IN, wherein the one or more components further serve to: determine, using the one or more machine learning models and based on at least the input data, one or more classifications that are assigned to the one or more second points, wherein the representation is further generated based on at least the one or more classifications.

[0247] P: Data center according to one of the paragraphs IO, wherein the one or more components further serve to: determine, using the one or more machine learning models and based at least on the input data, a heatmap that is associated with the traffic feature, wherein the representation is further generated based at least on the heatmap.

[0248] Q: Data center according to any of paragraphs IP, wherein: the traffic feature contains a road marking as represented by the sensor data representation corresponding to the map; the one or more processors are further designed to determine, based at least on the sensor data representation, a type of marking associated with the road marking; and the map is further updated to indicate the type of marking.

[0249] R: Data center as defined in paragraph IQ, wherein the data center comprises or is used in conjunction with at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing light transport simulations; a system for performing collaborative content creation for 3D assets; a system that provides one or more cloud gaming applications; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more generative AI operations;a system for performing operations using one or more large language models (LLMs); a system for performing operations using one or more vision language models (VLMs); a system for performing operations using one or more multimodal language models; a system for performing one or more conversational AI operations; a system for generating synthetic data; a system for presenting at least one type of virtual reality, augmented reality, or mixed reality content; systems that implement one or more multimodal language models; systems that use or employ one or more inference microservices; systems that involve deploying one or more machine learning models in a service or microservice along with an OS-level virtualization package (e.g., a container);a system that includes one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system that is at least partially implemented using cloud computing resources.

[0250] S: One or more processors, comprising: processing circuitry to generate a line representation associated with a traffic feature as represented by a map, wherein the line representation is generated based on at least the following: one or more encoders of one or more machine learning models generating one or more input tokens associated with one or more first points of the traffic feature and one or more image embeddings associated with an image of the traffic feature; and one or more decoders of the one or more machine learning models processing the one or more input tokens and the one or more embeddings to determine one or more second points associated with the line representation.

[0251] T: The one or more processors according to paragraph S, wherein the one or more processors comprise at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing light transport simulations; a system for performing collaborative content creation for 3D assets; a system that provides one or more cloud gaming applications; a system for performing one or more deep learning operations; a system that is implemented using an edge device; a system that is implemented using a robot; a system for performing one or more generative AI operations;a system for performing operations using one or more large language models (LLMs); a system for performing operations using one or more vision language models (VLMs); a system for performing operations using one or more multimodal language models; a system for performing one or more conversational AI operations; a system for generating synthetic data; a system for presenting at least one type of virtual reality, augmented reality, or mixed reality content; systems implementing one or more multimodal language models; systems using or employing one or more inference microservices; systems involving the deployment of one or more machine learning models in a service or microservice along with an OS-level virtualization package (e.g., a container);a system that includes one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system that is at least partially implemented using cloud computing resources.

[0252] It is understood that the aspects and embodiments described above are purely exemplary and that modifications of details may be made within the scope of protection of the claims.

[0253] Each device, each method and each feature disclosed in the description, and (where applicable) the claims and drawings, may be provided independently or in any suitable combination.

[0254] Reference numerals appearing in the claims are for illustrative purposes only and do not restrict the scope of protection of the claims. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] US 16 / 101,232

[0133] Cited non-patent literature

[0000] Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles” of the Society of Automotive Engineers (SAE) (Standard No. J3016-201806, published on June 15, 2018, Standard No. J3016-201609, published on September 30, 2016

[0091] Universal Mobile Telecommunications System, “UMTS”), Global System for Mobile Communication (Global System for Mobile Communication, “GSM”), IMT-CDMA Multi-Carrier (“CDMA2000

[0098]

Claims

[1] Procedure, encompassing: Generating one or more input tokens corresponding to one or more initial points associated with a road marker, as depicted by an image associated with a map; Generating one or more embeds that are associated with the image; Generating one or more machine learning models and, based on at least one or more input tokens and one or more embeddings, one or more output tokens corresponding to one or more second points associated with the road marking; Generating a line representation of the road marking based on at least one or more second points; and Update the map, based at least on the line representation, to include a label associated with the road marking. [2] The method of claim 1, further comprising at least one of the following: Receiving input data corresponding to one or more of the first points associated with the road marking; or Determine, based on at least the analysis of at least one of the first points from the map or image, which are associated with the road marking. [3] Method according to claim 1 or 2, wherein generating the one or more output tokens comprises: Generate, using one or more machine learning models and based on at least one or more input tokens and one or more embeddings, one or more first output tokens corresponding to a first section of one or more second points; and Generate, using one or more machine learning models and based on at least one or more first output tokens, one or more second output tokens corresponding to a second section of one or more second points. [4] Method according to any one of the preceding claims, further comprising: Generate, using one or more machine learning models and based on at least one or more input tokens and one or more embeddings, one or more image tokens associated with the image, where the generation of the line representation is furthermore based on at least one or more image tokens. [5] Method according to any one of the preceding claims, further comprising: Appending one or more input tokens to one or more learnable tokens to generate one or more appended input tokens, wherein the generation of the one or more input tokens is further based on at least the one or more attached input tokens and the one or more embeddings. [6] Method according to any one of the preceding claims, further comprising: Determine, based on at least one or more input tokens, one or more classifications that are assigned to one or more second points, where the generation of the line representation is furthermore based on at least one or more classifications. [7] Method according to any one of the preceding claims, further comprising: Generate, using one or more decoders and based on at least one or more input tokens, one or more coordinates that are assigned to one or more second points within the image, where the generation of the line representation is based on at least one or more coordinates. [8] Method according to any one of the preceding claims, further comprising: Generate, based on at least one or more output tokens or one or more image tokens associated with an image, a heatmap associated with the road marking, where the generation of the line display is furthermore based at least on the heatmap. [9] Data center, comprehensive: one or more central processing units (CPUs); one or more graphics processing units (GPUs); one or more isolated trusted execution environments (TEEs); one or more connections for multi-GPU communication; one or more data processing units (DPUs); one or more network interface chips (NICs); where one or more components of the data center serve the following purposes: Determining one or more initial points that are assigned to a traffic feature from a sensor data representation that corresponds to a map; Determine, using one or more machine learning models and based on at least input data associated with the one or more first points and the sensor data representation, the one or more second points associated with the traffic feature; Generating a representation of the traffic feature based on at least one or more of the second points; and Updating the map, based at least on the display, to include information associated with the traffic feature. [10] System according to claim 9, wherein one or more components further serve to: Generating one or more input tokens based on at least one or more initial points and one or more embeddings based on at least the sensor data representation, where the input data is assigned to one or more input tokens and one or more embeddings. [11] Data center according to claim 10, wherein one or more components further serve to: Appending one or more input tokens to one or more learnable tokens to generate one or more appended input tokens, where the input data is assigned to the one or more attached input tokens and the one or more embeddings. [12] Data center according to one of claims 9-11, wherein the determination of one or more second points associated with the traffic characteristic comprises: Generate, using one or more machine learning models and based on at least the input data, one or more output tokens; and Determine, based on at least one or more output tokens, one or more second points that are associated with the traffic characteristic. [13] Data center according to any one of claims 9 to 12, wherein one or more components further serve to perform at least one of the following: Receiving one or more inputs representing the first one or more points associated with the traffic characteristic; or Determine, based on at least the analysis of at least one of the map or sensor data representations, the one or more first points associated with the traffic feature. [14] Data center according to one of claims 9-13, wherein determining one or more second points associated with the traffic characteristic comprises: Determine, using one or more machine learning models and based on at least the input data, at least one first section of one or more second points; and Determine, using one or more machine learning models and based on at least two input data assigned to the at least first section of one or more second points, a at least second section of one or more second points. [15] Data center according to any of claims 9-14, wherein one or more components further serve to: Determine, using one or more machine learning models and based on at least the input data, one or more classifications that are assigned to one or more second points, where the representation is furthermore generated based on at least one or more classifications. [16] Data center according to any of claims 9-15, wherein one or more components further serve to: Determine, using one or more machine learning models and based on at least the input data, a heatmap that is assigned to the traffic feature, where the representation is furthermore generated based on at least the heatmap. [17] Data center according to any one of claims 9 to 16, wherein: The traffic feature contains a road marking, as represented by the sensor data representation corresponding to the map; one or more processors are further designed to determine, based at least on the sensor data representation, a type of marking that is associated with the road marking; and The map will be updated further to indicate the type of marker. [18] Data center according to any one of claims 9 to 17, wherein the data center comprises at least one of the following or is used in conjunction with one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing light transport simulations; a system for conducting collaborative content creation for 3D assets; a system that provides one or more cloud gaming applications; a system for performing one or more deep learning operations; a system implemented using an edge device; a system that is implemented using a robot; a system for performing one or more operations using generative AI; a system for performing operations using one or more large language models (LLMs); a system for performing operations using one or more Vision Language Models (VLMs); a system for performing operations using one or more multimodal language models; a system for performing one or more operations using conversational AI; a system for generating synthetic data; a system for presenting at least one type of virtual reality content, augmented reality content, or mixed reality content; Systems that implement one or more multimodal language models; Systems that use or employ one or more inference microservices; Systems that involve the deployment of one or more machine learning models in a service or microservice together with a virtualization package (e.g., a container) at the OS level; a system that includes one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system that is implemented at least partially using cloud computing resources. [19] One or more processors, comprising: Processing circuits to generate a line representation that is associated with a traffic feature as represented by a map, wherein the line representation is generated based on at least the following: one or more encoders of one or more machine learning models that generate one or more input tokens associated with one or more first points of the traffic feature, and one or more image embeddings associated with an image of the traffic feature; and one or more decoders of one or more machine learning models that process the one or more input tokens and the one or more embeddings to determine one or more second points associated with the line representation. [20] The one or more processors according to claim 19, wherein the one or more processors comprise at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing light transport simulations; a system for conducting collaborative content creation for 3D assets; a system that provides one or more cloud gaming applications; a system for performing one or more deep learning operations; a system that is implemented using an edge device; a system that is implemented using a robot; a system for performing one or more operations using generative AI; a system for performing operations using one or more large language models (LLMs); a system for performing operations using one or more Vision Language Models (VLMs); a system for performing operations using one or more multimodal language models; a system for performing one or more operations using conversational AI; a system for generating synthetic data; a system for presenting at least one type of virtual reality content, augmented reality content, or mixed reality content; Systems that implement one or more multimodal language models; Systems that use or employ one or more inference microservices; Systems that involve the deployment of one or more machine learning models in a service or microservice together with a virtualization package (e.g., a container) at the OS level; a system that includes one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system that is implemented at least partially using cloud computing resources.

Images