Associating traffic control devices with lanes for autonomous or semi-autonomous systems and applications

Abstract

The present disclosure relates to associating traffic control devices with lanes for autonomous or semi-autonomous systems and applications, in various examples, machine learning models may be trained and used to determine associations between traffic control devices (e.g., traffic signs, traffic lights, etc.) and lane segments of a driving road surface. According to the system and the method, a rule-based method and a machine learning-based method can be efficiently combined, and the system and the method are used for association of traffic control equipment and lanes. For example, training data may be synthetically generated based on traffic rules related to placement of traffic lights, and a machine learning model may be trained using the training data having a ground truth generated by the rules to associate traffic lights with respective lanes. Thus, the machine learning model may not require image data to predict a light-to-lane association. Different, given a non-image feature set indicative of lane segment and traffic light geometry and / or semantics, the machine learning model may predict associated lane segments for each traffic light.

Description

Linking traffic control devices to lanes for autonomous or semi-autonomous systems and applications. Background Technology

[0001] For autonomous or semi-autonomous vehicles to navigate safely in their environment, they may sometimes need to be able to identify which traffic control signals correspond to which lanes. This ability helps ensure the vehicle understands the current state of traffic control signals and their relevance to the vehicle's specific location and intended path, as well as which traffic controls correspond to other vehicles in the area. For example, misunderstanding traffic lights can lead to unsafe maneuvers, such as continuing to cross a red light into another lane or misjudging the timing of a turn. By effectively identifying the association between traffic signals and their corresponding lanes, vehicles can make informed decisions, ensuring adherence to traffic rules and enhancing overall safety in complex driving scenarios.

[0002] Generally, existing methods for associating traffic lights with lanes can be divided into two main categories: rule-based methods and machine learning-based methods. Rule-based methods typically rely on handcrafted heuristics derived from traffic rules and observations, allowing for rapid updates as new edge cases arise. However, these heuristics can be limited by the number of observations and may lead to errors in unfamiliar scenarios. Additionally, as the diversity of scenarios increases, the number of necessary heuristics also increases, complicating code maintenance.

[0003] On the other hand, machine learning methods can handle massive amounts of training data, covering a wider range of scenarios than developers typically encounter. However, a significant drawback of these methods is the high cost of collecting and labeling such large amounts of data, as it may require human annotators to carefully review each sample. These methods are typically based on pixel-derived image features. They may also struggle to meet the demand for large amounts of training data to capture variations in scene appearance, and they may also struggle to handle the challenges of representing the many possible combinations of traffic light placement and lane configuration. Summary of the Invention

[0004] Embodiments of this disclosure relate to associating traffic control devices with lanes for autonomous or semi-autonomous systems and applications. The disclosed systems and methods can be used to train and utilize machine learning models (e.g., deep neural networks) to determine the association between traffic control devices (e.g., traffic signs, traffic lights, etc.) and lane segments on a driving surface. The systems and methods of this disclosure can efficiently combine rule-based and machine learning-based approaches for associating traffic control devices with lanes. For example, training data can be synthetically generated based on traffic rules associated with the placement of traffic lights, and the machine learning model can be trained using training data with ground truth values ​​generated by the rules to associate traffic lights with corresponding lanes. Therefore, the machine learning model may not require image data to predict the association between lights and lanes. Instead, given a set of non-image features indicating the geometry and / or semantics of lane segments and traffic lights, the machine learning model can predict the associated lane segment for each traffic light.

[0005] Compared to conventional systems, in some embodiments, the systems disclosed herein are capable of training machine learning models to accurately predict the association between traffic control devices and lanes in unknown scenarios without modifying hard-coded rules when encountering new scenarios. Additionally, compared to conventional systems, the systems disclosed herein can be easily maintained by managing a single machine learning model rather than managing complex logic across a target deployment platform. For example, during training, rules may be extended to highly complex scenarios, while in deployment, the system can include a single, easily maintainable learned model. Furthermore, the systems disclosed herein are capable of training models to associate traffic control devices with lanes without using image data as input. Therefore, these systems are capable of synthesizing training data based on traffic rules governing traffic light placement at intersections and other scenarios, and generating ground truth based on those rules. Attached Figure Description

[0006] The system and method for associating traffic control equipment with lanes for use in autonomous or semi-autonomous systems and applications are described in detail below with reference to the accompanying drawings, wherein:

[0007] Figure 1 is a data flow diagram illustrating an example of a process for associating traffic control equipment with lane segments according to some embodiments of the present disclosure;

[0008] Figure 2 illustrates an example of a scenario in which the association between a traffic control device and a lane can be determined according to some embodiments of the present disclosure;

[0009] Figure 3 is a data flow diagram illustrating an example of the process by which a machine learning model according to some embodiments of the present disclosure associates traffic control signals with lanes in a scenario similar to that of Figure 2;

[0010] Figure 4 illustrates another example scenario in which the association between traffic control equipment and lanes can be determined according to some embodiments of the present disclosure;

[0011] Figure 5 is a data flow diagram illustrating another example of a machine learning model according to some embodiments of the present disclosure performing the association of traffic control signals with lanes in a scenario similar to that of Figure 4;

[0012] Figure 6 is a data flow diagram illustrating an example process for training one or more machine learning models to determine the association between traffic control equipment and corresponding lanes, according to some embodiments of the present disclosure;

[0013] Figure 7 illustrates an example of a system that can perform one or more processes described herein, according to some embodiments of this disclosure;

[0014] Figure 8 is a flowchart illustrating an example of a method for associating an activated traffic control signal with a lane segment according to some embodiments of the present disclosure;

[0015] Figure 9 is a flowchart illustrating an example of a method for associating traffic control equipment with a lane according to some embodiments of the present disclosure;

[0016] Figure 10 is a flowchart illustrating an example of a method for synthetically generating data for training a machine learning model to determine the association between traffic control equipment and lanes, according to some embodiments of the present disclosure.

[0017] Figure 11A is an illustration of an example autonomous vehicle according to some embodiments of the present disclosure;

[0018] Figure 11B is an example of the camera position and field of view of an example autonomous vehicle of Figure 11A according to some embodiments of the present disclosure;

[0019] Figure 11C is a block diagram of an example system architecture of an example autonomous vehicle of Figure 11A according to some embodiments of the present disclosure;

[0020] Figure 11D is a system diagram of communication between a cloud-based server and the example autonomous vehicle of Figure 11A, according to some embodiments of the present disclosure.

[0021] Figure 12 is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure; and

[0022] Figure 13 is a block diagram of an example data center suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0023] Systems and methods for associating traffic control devices with lanes for autonomous and semi-autonomous systems and applications are disclosed. While this disclosure may be described with reference to example autonomous or semi-autonomous vehicles or machines 1100 (which may be alternatively referred to herein as “vehicle 1100,” “self-vehicle 1100,” “self-machine 1100,” or “machine 1100,” examples of which are described with reference to Figures 11A through 11D), this is not intended to be limiting. For example, the systems and methods described herein may be used by, but are not limited to, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, airships, boats, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, underwater vehicles, drones, and / or other vehicle types. Furthermore, while this disclosure may be described with respect to associating traffic lights with lanes for autonomous or semi-autonomous navigation, this is not intended to be limiting. The systems and methods described herein can be used in augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications, and / or can be used in any other technical field that associates traffic control annotations (e.g., traffic lights, traffic signs) with lanes.

[0024] Generally, training DNNs or other machine learning models to associate traffic control devices with lanes typically requires a large amount of labeled data to cover various scenarios. In some examples, to reduce the need for human-labeled data, the system disclosed herein decouples the problem of associating traffic control devices with lanes from the problems of traffic control device detection and classification, as well as lane detection and lane direction classification. This allows the system to infer the ground truth of traffic control device-lane association through rule-based reasoning. At least based on this decoupling of these problems, the system disclosed herein can combine rule-based and machine learning-based methods to identify the association between traffic control devices and lanes.

[0025] For example, the system disclosed herein can use rules to generate pseudo-ground values ​​to train a machine learning model. In such an example, it may not be necessary to manually label the association between traffic control devices and lanes in each frame, and training data can be generated purely based on traffic rules regarding the placement of traffic control devices (e.g., traffic lights, traffic signs, etc.). Then, in some cases, purely synthetic data with ground values ​​generated by the rules can be used to train a DNN or other type of machine learning model for determining the association between traffic control devices and lanes. In some examples, such synthetic data may only need to contain metadata associated with lanes and traffic control devices. That is, synthetic data may not need to include synthetic image data. While all or most of the training data may come from synthetic sources, in some cases, the training data may be supplemented by real data labeled by humans for new scenarios. Once the model is trained, the trained model is able to identify the association between traffic control devices and lanes in various scenarios without requiring image data or image features as input.

[0026] For example, one or more systems may use one or more sensors of the machine to acquire sensor data. In some examples, sensor data may include LiDAR data generated using one or more LiDAR sensors, RADAR data generated using one or more RADAR sensors, image data generated using one or more image sensors (e.g., one or more cameras), ultrasonic data generated using one or more ultrasonic sensors, or any other type of sensor data. Sensor data may be applied to one or more perception systems or components. For example, sensor data may be applied to lane perception systems and / or traffic control device perception systems. In various examples, lane perception systems may be configured to generate various geometric and / or semantic features associated with lanes and / or lane segments in the machine's operating environment. Similarly, traffic control device perception systems may be configured to generate various geometric and / or semantic features corresponding to traffic control devices in the environment, such as the location of the traffic control devices, the state associated with the traffic control devices, the geometry of the traffic control devices, or any other features.

[0027] As described herein, in some examples, lane features and / or traffic control device features may include either non-image features or non-pixel features. In other words, while features can be derived by processing images or some other form of visual or sensor data (e.g., point clouds, detection points, etc.), the features themselves may not include pixels, portions of an image, or other visual features. For example, features of the location and / or size of traffic control devices may be represented as bounding boxes in 2D images and / or 3D space. As another example, features corresponding to the geometry of a lane segment may be represented by the centerline, left boundary, right boundary, start point, and end point of the lane segment, and this geometry may also be represented in 2D images and / or 3D spatial coordinate systems.

[0028] In some examples, lane and traffic control device features can be used as input data to apply to one or more machine learning models (e.g., one or more DNNs) that can be trained to predict the association between traffic control devices and lanes. For example, given a set of input features including lane segment geometry, lane segment orientation, traffic light geometry (e.g., location, size, orientation), expected road users, and state, one or more machine learning models can output a classification of the pairing of active traffic light bulbs with lane segments. In some cases, such as for the association of traffic lights with lanes on self-contained roads at intersections, one or more machine learning models can directly predict the associated lane orientation for each active traffic light bulb. Additionally or alternatively, one or more machine learning models can predict the associated lane orientation of traffic control devices (e.g., signs, lights, etc.).

[0029] In some cases, one or more machine learning models may include one or more embedding layers. These embedding layers can project lane features and traffic control device features into an embedding space before feeding them into one or more backbone layers. Depending on the type of input features, one or more machine learning models may use different embedding layers to perform the projection. For example, for discrete values ​​(e.g., discrete lane directions and discrete traffic light bulb states), one or more machine learning models may use lookup tables, which may be hard-coded or learned. As another example, for continuous values ​​(e.g., 2D coordinates of lane centerlines and 2D coordinates of traffic lights), one or more machine learning models may use multilayer perceptrons. In some examples, one or more machine learning models may expand these embeddings to a specific size (e.g., NxM) in the first two dimensions, and lane segment embeddings and traffic light embeddings may be concatenated for use by the machine learning backbone. For example, an NxK' embedding can be copied M times to form an NxMxK' embedding, and an MxL' embedding can be copied N times to form an NxMxL' embedding. These expanded embeddings can then be concatenated to form an NxMx(K'+L') embedding before being applied to one or more backbone layers. In some examples, one or more backbone layers may include one or more of a multilayer perceptron, convolutional neural network, transformer, recurrent neural network, graph neural network, and / or combinations thereof, or any other type of neural network.

[0030] As described herein, in some cases, features corresponding to one or more traffic control devices and / or one or more lane segments can be applied as input vectors to one or more machine learning models. For example, a first number (e.g., “N”) of K-dimensional vectors can be applied to one or more machine learning models, where each K-dimensional vector can represent the geometry and semantics of a lane segment. Additionally, a second number (e.g., “M”) of L-dimensional vectors can be applied to one or more machine learning models, where each L-dimensional vector can represent the geometry and semantics of a traffic control device (e.g., an activated traffic light bulb). In such an example, one or more machine learning models can output an NxM matrix, where each entry in the matrix can include a number representing the confidence that the Nth lane segment and the Mth traffic control device (e.g., an activated traffic light bulb) are a valid pairing.

[0031] In some examples, one or more machine learning models can output a binary classification of the validity of each pair of traffic control devices and lane segments in a scenario. For example, one or more machine learning models can output a binary matrix where each entry corresponds to a pair of lane segments and traffic control devices in the scenario and indicates whether the pair is valid. Additionally or alternatively, one or more machine learning models can output a binary classification of the lane direction associated with each traffic control device (e.g., each active traffic light bulb). For example, when considering the association between active traffic light bulbs and lanes on a lane segment on a self-owned road at an intersection, and the lane segment direction for each lane segment on the self-owned road is known, one or more machine learning models can actually simplify the problem of classifying the lane direction corresponding to each active traffic light bulb. For example, one or more first active traffic light bulbs can be used for one or more first lane directions for left turns and / or left U-turns, one or more second active traffic light bulbs can be used for one or more second lane directions for going straight and right turns, and so on.

[0032] In any example, one or more systems may use one or more machine learning models to compute multiple confidence scores, at least based on input data. These confidence scores indicate whether one or more activation signals from multiple traffic control devices correspond to one or more lane segments. Using the confidence scores, one or more systems may associate one or more activation signals with one or more lane segments. In some examples, one or more systems may enable a machine (e.g., an autonomous or semi-autonomous machine or vehicle) to perform one or more actions based at least on the associations between one or more activation signals and one or more lane segments. For example, one or more systems may use these associations to plan a path for the machine to follow, adjust the machine's speed (e.g., brake or stop if the lane light is red, maintain speed if the lane light is green, etc.), or perform any other actions described herein.

[0033] In some examples, one or more systems may use synthetically generated training data to train one or more machine learning models. In some cases, one or more systems may use one or more rules associated with the positioning of traffic control signaling devices relative to one or more corresponding lane segments to synthetically generate training data (e.g., training inputs and / or ground truth data) from non-visual data.

[0034] For example, at intersections where there may be more lanes than traffic lights, and where there may be multiple lane directions, one or more systems can automatically and synthetically generate ground-value associations between traffic lights and lanes without using image information. To this end, one or more systems can generate 3D lane geometry based on traffic rules (e.g., specifying an effective lane width range) and given a random seed for the number of lanes. In some cases, one or more systems can project the 3D lane geometry onto a 2D image space using the camera's intrinsic and extrinsic parameters to obtain a 2D geometry. For each lane segment, one or more systems can generate lane directions based on traffic rules (e.g., in right-hand drive countries, the right-turn lane should be the rightmost lane). Additionally, one or more systems can generate 3D traffic light geometry based on traffic rules (e.g., specifying effective traffic light placement and size on the lanes) and given a random seed for the number of lights. Using the camera's intrinsic and extrinsic parameters, one or more systems can project the 3D geometry of the traffic lights onto a 2D image space to obtain a 2D geometry. For each traffic light, one or more systems can generate target road users and states based on traffic rules. For example, for a traffic light bulb with multiple arrows (e.g., left arrow and straight arrow), one or more systems may represent it as multiple arrow states. Then, for each active traffic light bulb, one or more systems may assign its associated lane segment based on traffic rules in a priority order. Examples of such priority order may include, but are not limited to (e.g., other combinations or orders are also possible), (1) a light for road users other than vehicles is not associated with any lane segment; (2) a light is not facing its own road (from the direction) or is not at the current intersection (from the 3D geometry); (3) if a U-turn arrow bulb is present, it controls only the U-turn lane; (4) if a left arrow bulb is present, it controls the left-turn lane; if a U-turn bulb is not present, it also controls the left U-turn lane; (5) if a right arrow bulb is present, it controls the right-turn lane; (6) if a solid circle bulb is present, it controls the remaining lanes; (7) if a straight arrow bulb is present, it controls the straight lane.

[0035] As another example, in the case of a highway entrance ramp, where corresponding lights control each lane, one or more systems can automatically and synthetically generate ground truth traffic light-lane associations without using image information. To this end, one or more systems can generate 3D lane geometry based on traffic rules (e.g., specifying an effective lane width range) and given a random seed for the number of lanes. In some cases, one or more systems can project the lane 3D geometry onto a 2D image space using camera intrinsic and / or extrinsic parameters to obtain a 2D geometry. For each lane segment, one or more systems can generate lane direction based on traffic rules (e.g., in a right-hand drive country, the right-turn lane should be the rightmost lane). Additionally, one or more systems can generate 3D traffic light geometry based on traffic rules (e.g., specifying effective traffic light placement and traffic light size on the lanes) and given a random seed for the number of lights. Using camera intrinsic and / or extrinsic parameters, one or more systems can project the 3D geometry of the traffic lights onto a 2D image space to obtain a 2D geometry. For each traffic light, one or more systems can generate target road users and states, represented as one of red, yellow, and green circles. Then, for each activated traffic light bulb, one or more systems can assign its associated lane segment based on the traffic light's proximity to the nearest lane segment in 3D.

[0036] In some examples, during training, one or more systems may apply sigmoid activation and use binary cross-entropy loss. However, during training, any other loss function compatible with binary cross-entropy (such as focus loss) or uncertainty loss with a corresponding activation function, label smoothing, and any other regularization method may be applied. To train one or more machine learning models, one or more systems may apply synthetically generated training inputs to one or more machine learning models and then compute the loss between the outputs of one or more machine learning models and the synthetically generated ground truth data. Based on these losses, one or more systems may update one or more parameters (e.g., weights, biases, etc.) of one or more machine learning models to minimize or otherwise reduce the loss.

[0037] In some embodiments, the systems and methods described herein can be performed in a simulated environment (e.g., NVIDIA's DriveSIM) using simulated data (e.g., simulated sensor data from simulated sensors of virtual machines or simulated machines). For example, simulated input data (e.g., perception data, non-image features, or any other data described herein) can be applied to one or more machine learning models to determine the association between traffic control devices and lanes in the simulated environment, and this information can be used to perform operations associated with virtual machines in the simulated environment. These simulated operations can be used to test their performance before deploying the underlying algorithms, systems, and / or processes described herein to the real world.

[0038] In any example, such as in a simulated environment used for testing, validation, training, etc., the simulated environment may be rendered or otherwise generated using one or more optical transport algorithms (such as ray tracing and / or path tracing algorithms). In some embodiments, the simulated environment and / or one or more of its objects, features, or components may be generated or managed within a 3D content collaboration platform (e.g., NVIDIA's OMNIVERSE) for industrial digitization, generative physics AI, and / or other use cases, applications, or services. For example, the content collaboration platform or system may include systems for using or developing generic scene descriptors (USD) (e.g., OpenUSD) data to manage objects, features, scenes, etc., in simulated environments, digital environments, etc. The platform may include realistic physics simulations, such as using NVIDIA's PhysX SDK, to simulate real physics and physical interactions with simulations hosted on the platform. This platform integrates OpenUSD with ray tracing / path tracing / optical transport simulations (e.g., NVIDIA's RTX rendering technology) into software tools and simulation workflows for building, training, deploying, or testing AI systems, such as systems used for testing, validating, training (e.g., machine learning models, neural networks, etc.) and / or other tasks related to automobiles, robots, machines, or other applications. In some examples, the simulated environment can include digital twins of real-world environments, such as specific road sections, warehouses, data centers, airports, geographic areas, ocean areas, or any other real-world environment where autonomous or semi-autonomous machines can operate.

[0039] In some embodiments, a remote control or teleoperation system can be used to perform remote operation or remote control of a vehicle or other machine. For example, the systems and methods described herein can be used to identify associations between traffic control signals and lanes, which can be included in a visualization or mapping of the environment to help a remote operator control (or provide waypoints or other control or navigation instructions) autonomous or semi-autonomous machines through the environment.

[0040] In some examples, one or more machine learning models described herein (e.g., deep neural networks, language models, LLM, VLM, multimodal language models, perceptual models, tracking models, fusion models, transformer models, diffusion models, encoder-only models, decoder-only models, encoder-decoder models, neural rendering field (NERF) models, etc.) can be packaged into microservices (such as inference microservices (e.g., NVIDIA NIM)), which may include containers (e.g., operating system (OS)-level virtualization packages) that may include an application programming interface (API) layer, a server layer, a runtime layer, and / or a model "engine". For example, an inference microservice may include the container itself and one or more models (e.g., weights and biases). In some cases, such as when one or more machine learning models are small enough (e.g., have a sufficiently small number of parameters), one or more models may be contained within the container itself. In other examples, such as when one or more models are large, one or more models may be hosted / stored in the cloud (e.g., in a data center) and / or may be hosted on-premises and / or at the edge (e.g., on a local server or computing device, but outside the container). In these embodiments, one or more models can be accessed through one or more APIs (such as REST APIs). Therefore, in some embodiments, one or more machine learning models described herein can be deployed as inference microservices to accelerate the deployment of one or more models on any cloud, data center, or edge computing system while ensuring data security. For example, an inference microservice may include one or more APIs, pre-configured containers for simplified deployment, optimized inference engines (e.g., execution software built using standardized AI model deployments, such as NVIDIA's Triton Inference Server, and / or one or more APIs for high-performance deep learning inference, which may include inference runtimes and model optimizations for providing low latency and high throughput for production applications, such as NVIDIA's TensorRT), and / or enterprise management data for telemetry (e.g., including identity, metrics, health checks, and / or monitoring). One or more machine learning models described herein, along with acceleration infrastructure, may be included as part of a microservice, capable of deployment using a single command and / or orchestrated and automatically scaled using a container orchestration system on the acceleration infrastructure (e.g., up to data center scale on a single device). Therefore, inference microservices may include one or more machine learning models (e.g., optimized for high-performance inference), inference runtime software that executes one or more machine learning models and provides output / response to inputs (e.g., user queries, prompts, etc.), and enterprise management software that provides health checks, identity and / or other monitoring.In some embodiments, the inference microservice may include software for performing in-situ replacements and / or updates to one or more machine learning models. When a replacement or update is performed, the software performing the replacement / update may maintain user configurations for the inference runtime software and the enterprise management software.

[0041] The systems and methods described herein can be used by, but are not limited to, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles connected to one or more trailers, aircraft, boats, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, underwater vehicles, drones, and / or other vehicle types. Furthermore, the systems and methods described herein can be used for a variety of purposes, including, but not limited to, machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and surveillance, simulation and digital twins, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or participant simulation and / or digital twins, data center processing, conversational artificial intelligence (AI), optical transmission simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation of 3D assets, cloud computing, and / or any other suitable application.

[0042] The disclosed embodiments can be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines), systems implemented using robots, aviation systems, medical systems, boating systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems implementing language models (such as large language models (LLM), visual language models (VLM), and / or multimodal language models), systems implementing one or more multimodal language patterns, systems using or deploying one or more inference microservices, systems combining and deploying one or more machine learning models in services or microservices and OS-level virtualization packages (e.g., containers), systems containing one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in a data center, systems for performing conversational AI operations, systems for performing optical transmission simulations, systems for performing collaborative content creation of 3D assets, systems for performing generative AI operations, systems implemented at least partially using cloud computing resources, and / or other types of systems.

[0043] Referring to Figure 1, Figure 1 illustrates an example data flow diagram of process 100 for associating traffic control equipment with lane segments according to some embodiments of the present disclosure. It should be understood that such and other arrangements described herein are presented only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or in lieu of the arrangements and elements shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in combination with other components, and may be implemented in any suitable combination and location. The various functions performed by the entities described herein may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory. In some embodiments, the systems, methods, and processes described herein may be performed using components, features, and / or functions similar to the example autonomous vehicle 1100 of Figures 11A-11D, the example computing device 1200 of Figure 12, and / or the example data center 1300 of Figure 13.

[0044] The process 100 shown in the example of Figure 1 can be implemented using additional or alternative components such as one or more sensors 102, lane sensors 104, traffic control device sensors 106, one or more machine learning models 108, association components 110, and one or more driving stack components 112. As a brief overview of process 100, one or more sensors 102 can generate sensor data 114, which is used as input to lane sensors 104 and traffic control device sensors 106. Using sensor data 114, lane sensors 104 can generate one or more lane features 116, and traffic control device sensors 106 can generate one or more traffic control device features 118. One or more machine learning models 108 can obtain and use one or more lane features 116 and one or more traffic control device features 118 to calculate one or more confidence scores 120, which can represent one or more probabilities that the association between a traffic control device and a lane is a valid pairing. The association component 110 can use one or more confidence scores 120 to determine the association 122 between one or more traffic control devices and lanes. The association 122 between one or more traffic control devices and lanes can then be provided to one or more driving stack components 112, which can use the one or more associations to control one or more operations of the machine.

[0045] In some examples, one or more sensors 102 may include any or more sensors of the autonomous vehicle 1100. For example, one or more sensors 102 may include one or more LiDAR sensors, one or more RADAR sensors, one or more image sensors (e.g., one or more cameras), one or more ultrasonic sensors, or one or more of any other sensors described herein. Therefore, sensor data 114 may include LiDAR data generated using one or more LiDAR sensors, RADAR data generated using one or more RADAR sensors, image data generated using one or more image sensors, ultrasonic data generated using one or more ultrasonic sensors, or any other type of sensor data. Sensor data 114 may be applied to one or more perception systems or components. For example, sensor data 114 may be applied to lane perceiver 104 and traffic control device perceiver 106. In various examples, lane perceiver 104 may be configured to generate one or more lane features 116, which may include various geometric and / or semantic features (e.g., lane direction) associated with lanes and / or lane segments in the machine's operating environment. Similarly, the traffic control device sensor 106 can be configured to generate one or more traffic control device features 118, which may include various geometric and / or semantic features corresponding to traffic control devices in the environment, such as the location of the traffic control device, the state associated with the traffic control device, the geometry of the traffic control device, or any other features.

[0046] As described herein, in some examples, one or more lane features 116 and / or one or more traffic control device features 118 may include either non-image features or non-pixel features. In other words, while features can be derived by processing image data or some other form of visual data, the features themselves may not include pixels, portions of an image, or other visual features. For example, features of the location and / or size of a traffic control device may be represented as bounding boxes in 2D and / or 3D image space. As another example, features corresponding to the geometry of a lane segment may be represented by the centerline, left boundary, right boundary, start point, and end point of the lane segment, and this geometry may also be represented in a 2D and / or 3D image space coordinate system.

[0047] As illustrated in Figure 1, one or more lane features 116 and one or more traffic control device features 118 can be input to one or more machine learning models 108, which can be trained to predict the association between traffic control devices and lanes. For example, given a set of input features indicating lane segment geometry, lane segment orientation, traffic control device geometry (e.g., location, size, orientation), expected road users, and state, one or more machine learning models 108 can output one or more confidence scores 120 for one or more classifications indicating the pairing of traffic control devices with lane segments. In some cases, such as for the association between activated traffic lights and lanes on a self-contained road at an intersection, one or more machine learning models 108 can directly predict the associated lane orientation for each activated traffic light bulb.

[0048] While this article describes examples of using neural networks, particularly DNNs, in machine learning models, this is not intended to be limiting. Any of the various machine learning models and / or neural networks described herein can include any type of machine learning model, such as one or more machine learning models using linear regression, logistic regression, decision trees, support vector machines (SVM), Naive Bayes, k-nearest neighbors (Knn), k-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoder neural networks, artificial neural networks (ANN), convolutional neural networks (CNN), recurrent neural networks (RNN), perceptrons, long short-term memory (LSTM) networks, multilayer perceptron (MLP) networks, deep stacked networks (DSN), generative pre-trained (GPT) models or networks, feedforward networks, radial basis function ANNs, self-organizing networks, etc. Sequence-to-sequence models include Sequence-to-Sequence (SOM), Kohonen mapping, Hopfield networks, Boltzman machines, deep belief neural networks, deconvolutional neural networks, generative adversarial networks (GAN), liquid state machines, modular neural networks, sequence-to-sequence models, networks using transformer architectures, diffusion models (e.g., diffusion probability models, score-based generative models, etc.), Neural Render Field (NeRF) models, models with encoder-only architectures, models with decoder-only architectures, patterns with encoder-decoder architectures, generative machine learning models, language models, large language models (LLM), visual language models (VLM), multimodal language models (MMLM), etc.) and / or other types of machine learning models.

[0049] In some cases, one or more machine learning models 108 may include one or more embedding layers. One or more embedding layers may project lane features 116 and traffic control device features 118 into the embedding space before applying them to one or more backbone layers. Depending on the type of input features, one or more machine learning models 108 may use different embedding layers to perform the projection. For example, for discrete values ​​(e.g., discrete lane directions and discrete traffic light bulb states), one or more machine learning models 108 may use lookup tables, which may be hard-coded or learned. As another example, for continuous values ​​(e.g., lane centerline 2D coordinates and traffic light 2D coordinates), one or more machine learning models 108 may use multilayer perceptrons. In some examples, one or more machine learning models 108 may expand these embeddings to a specific size (e.g., NxM) in the first two dimensions, and lane segment embeddings and traffic control device embeddings may be concatenated for use by the machine learning backbone. For example, an NxK' embedding can be copied M times to form an NxMxK' embedding, and an MxL' embedding can be copied N times to form an NxMxL' embedding. The expanded embeddings can then be concatenated to form an NxMx(K'+L') embedding before being applied to one or more backbone layers. In some examples, one or more backbone layers may include one or more of a multilayer perceptron, convolutional neural network, transformer, recurrent neural network, graph neural network, and / or combinations thereof, or any other type of neural network.

[0050] As described herein, in some cases, one or more lane features 116 and / or one or more traffic control device features 118 may be represented as vectors input to one or more machine learning models 108. For example, one or more lane features 116 may include a first number (e.g., “N”) of K-dimensional vectors, where each K-dimensional vector may represent the geometry and semantics of a lane segment. Additionally, one or more traffic control device features 118 may include a second number (e.g., “M”) of L-dimensional vectors, where each L-dimensional vector may represent the geometry and semantics of a traffic control device (e.g., an activated traffic light bulb). Additionally, in some examples, one or more confidence scores 120 output by one or more machine learning models 108 may include an NxM matrix, where each entry in the matrix may include a number or value representing the confidence that the Nth lane segment and the Mth traffic control device (e.g., an activated traffic light bulb) are a valid pairing.

[0051] In some examples, one or more confidence scores 120 output by one or more machine learning models 108 may include a binary classification of the validity of each pair of traffic control devices and lane segments in the scenario. For example, one or more confidence scores 120 may include a binary matrix where each entry corresponds to a paired lane segment and traffic control device in the scenario and indicates whether the pair is valid. That is, the value or number in each entry may represent the confidence level of whether the pair is valid. Additionally or alternatively, one or more machine learning models 108 may output a binary classification of the lane direction associated with each traffic control device (e.g., each active traffic light bulb). For example, when considering the association between active traffic light bulbs and lanes on a lane segment on a self-road at an intersection, and the lane segment direction for each lane segment on the self-road is known, one or more machine learning models 108 may actually simplify the problem of classifying the lane direction corresponding to each active traffic light bulb. For example, one or more first-activated traffic light bulbs may be used for one or more first lane directions for left turns and / or left U-turns, one or more second-activated traffic light bulbs may be used for one or more second lane directions for going straight and right turns, and so on.

[0052] In some examples, association component 110 may use one or more confidence scores 120 to generate one or more associations 122 between traffic control devices and lanes. Although this document describes the association of traffic control devices with lane segments, this is not intended to be limiting, and in addition to additional or replacement elements, one or more machine learning models 108 and / or association component 110 may determine the association between traffic control devices (e.g., traffic signs, traffic light fixtures / housings, etc.) and lane segments, the association between activated traffic control signals (e.g., illuminated traffic light bulbs) and lane segments, the association between activated traffic control signals and lane segment directions, etc. In some examples, association component 110 may associate traffic control devices (or traffic control signals, activated light bulbs, etc.) with lane segments based on confidence scores for pairing traffic control devices with lanes, such confidence scores being the highest confidence score, the highest confidence score exceeding a threshold, a confidence score having a value that meets or exceeds a threshold (e.g., 80% confidence, 90% confidence, etc.), or any other criterion or metric. For example, if the association component 110 determines, based on one or more confidence scores 120, that a first confidence score for a first pairing between a first traffic control device and a first lane segment is greater than a second confidence score for a second pairing between the first traffic control device and a second lane segment and / or a third confidence score for a third pairing between the second traffic control device and the first lane segment, then the association component 110 may determine that the first pairing is a valid pairing. Additionally, in some examples, in order for the association component 110 to determine that the first pairing is a valid pairing, the association component may determine that the first confidence score exceeds the second confidence score and / or the third confidence score by a threshold (e.g., 10% confidence, 20% confidence, 40% confidence, etc.).

[0053] In some examples, the association 122 between one or more traffic control devices and lanes can be sent or provided to one or more driving stack components 112. One or more driving stack components 112 can use the association 122 between one or more traffic control devices and lanes to cause the machine to perform one or more control operations. For example, one or more driving stack components can use the association 122 between one or more traffic control devices and lanes to plan a path for the machine, adjust the machine's speed (e.g., brake or stop if the lane light is red, maintain speed if the lane light is green, etc.), or perform any other control operation as described herein.

[0054] In various examples, one or more driving stack components 112 may include perception components, model components, planning components, control components, avoidance components, actuation components, waiting sensors, and / or other components corresponding to additional and / or alternative layers of the driving stack. These components may use the association 122 of one or more traffic control devices with the lane as input to make various decisions on behalf of the machine.

[0055] Waiting sensors can be responsible for determining constraints on a machine based on rules, conventions, and / or practical considerations. For example, rules, conventions, and / or practical considerations might relate to traffic lights, multi-intersection stopping, yielding, merging, toll plazas, gates, police or other emergency personnel, road workers, stopped buses or other machinery, one-way bridge arbitration, ferry entrances, etc. Therefore, waiting sensors can be used to identify potential obstacles and implement one or more controls (e.g., deceleration, stopping, etc.) that might not be possible with obstacle sensors alone.

[0056] The planning components may include a route planner, a lane planner, a behavior planner, and / or a behavior selector, as well as other components, features, and / or functions. The route planner may generate a planned route using information from one or more traffic control devices and their associations with lanes. This planned route may consist of GNSS waypoints (e.g., GPS waypoints), 3D world coordinates (e.g., Cartesian coordinates, polar coordinates, etc.) indicating coordinates relative to the origin on the machine. Waypoints may represent specific future distances for the machine, such as city blocks, kilometers, feet, inches, miles, etc., which can be used as targets for the lane planner.

[0057] The avoidance components of one or more driving stack components 112 can help the machine avoid collisions with objects (e.g., moving and stationary objects) and / or avoid traffic violations. The avoidance components can use one or more traffic control and lane associations 122 to determine whether the machine's intended behavior or path will violate traffic rules (e.g., running a red light, using an incorrect lane, etc.). In some examples, the avoidance components can ignore traffic rules, road rules, and courteous driving norms to ensure that no collision occurs between the machine and any object. Therefore, the obstacle avoidance layer can be a layer separate from the road layer rules, and the obstacle avoidance layer ensures that the machine performs safe actions only from an obstacle avoidance perspective. On the other hand, the road layer rules ensure that the machine complies with traffic rules and conventions and adheres to legal and regular right-of-way (as described herein).

[0058] Referring now to FIG2, FIG2 illustrates an example of a scenario in which the association of traffic control devices with lanes can be determined according to some embodiments of the present disclosure. Environment 200 may include a plurality of traffic control devices 202(1)-202(6) (hereinafter collectively referred to as “traffic control devices 202”) and a plurality of lanes 204(1)-204(4) (hereinafter collectively referred to as “lanes 204”). In this case, one or more systems of the present disclosure may associate one or more traffic control devices 202 with one or more lanes 204. In the example of FIG2, traffic control devices 202 correspond to traffic lights or traffic signals, and in some cases, one or more systems of the present disclosure may associate one or more active bulbs of traffic control device 201 with one or more lane directions associated with one or more lanes 204. For example, lane features corresponding to lane 204 and traffic control device features corresponding to traffic control device 202 can be applied to a deep neural network that can use these features to calculate confidence scores associated with possible pairings of traffic control device 202 and lane 204, and one or more traffic control devices 202 can be associated with one or more lanes 204 based on the confidence scores. For example, in the example of Figure 2, a first traffic control device 202(1) can be associated with a first lane 204(1), a second traffic control device 202(2) can be associated with a second lane 204(2) and a third lane 204(3). A third traffic control device 202(3) can be associated with a fourth lane 204(4). Additionally, in some cases, one or more systems can ignore traffic control devices 202(4)-202(6) because these traffic control devices may be used to control traffic in non-self lanes (e.g., oncoming traffic lanes).

[0059] For example, Figure 3 is a data flow diagram illustrating an example of a process 300 of a machine learning model according to some embodiments of the present disclosure for associating traffic control signals with lanes in a scenario similar to that shown in Figure 2. As illustrated in the example of Figure 3, one or more first embedding layers 302A may use one or more lane features 116 to generate one or more lane segment embeddings 304. Similarly, one or more second embedding layers 302B may use one or more traffic control device features 118 to generate one or more traffic control device embeddings 306. Although depicted as separate components, in some examples, one or more first embedding layers 302A and one or more second embedding layers 302B may be the same component. One or more lane segment embeddings 304 and one or more traffic control device embeddings 306 may be fed into one or more copy and stitch layers 308, and one or more copy and stitch layers 308 may generate one or more traffic control device and lane embeddings 310. The backbone and classification head 312 of one or more machine learning models 108 can use the embeddings 310 of one or more traffic control devices with lanes to compute one or more confidence scores 120 indicating the association between one or more traffic control devices and lanes.

[0060] Referring back to the example in Figure 2, in some examples, one or more systems can automatically and synthetically generate ground truth data indicating the association between traffic control equipment and lanes without using image information. To perform this operation on scenarios similar to those shown in the example of Figure 2 (e.g., involving more complex intersections), one or more systems can generate 3D lane geometry based on traffic rules (e.g., specifying an effective range of lane widths) and given a random seed for the number of lanes. In some cases, one or more systems can project the 3D lane geometry onto a 2D image space using the camera's intrinsic and extrinsic parameters to obtain a 2D geometry. For each lane segment, one or more systems can generate lane orientation based on traffic rules (e.g., in right-hand driving countries, the right-turn lane should be the rightmost lane). Additionally, one or more systems can generate 3D traffic light geometry based on traffic rules (e.g., specifying effective traffic light placement and size on the lanes) and given a random seed for the number of lights. Using the camera's intrinsic and extrinsic parameters, one or more systems can project the 3D geometry of the traffic lights onto a 2D image space to obtain a 2D geometry. For each traffic light, one or more systems can generate target road users and states based on traffic rules. For example, for a traffic light bulb with multiple arrows (e.g., left arrow and straight arrow), one or more systems can represent it as multiple arrow states. Then, for each active traffic light bulb, one or more systems can assign its associated lane segment according to traffic rules based on a priority order. Examples of such priority order may include, but are not limited to (e.g., other combinations or orders are also possible), (1) lights for road users other than vehicles are not associated with any lane segment; (2) lights are not facing their own road (from the direction) or are not at the current intersection (from the 3D geometry); (3) if a U-turn arrow bulb is present, it controls only the U-turn lane; (4) if a left arrow bulb is present, it controls the left-turn lane; if a U-turn bulb is not present, it also controls the left U-turn lane; (5) if a right arrow bulb is present, it controls the right-turn lane; (6) if a solid circle bulb is present, it controls the remaining lanes; (7) if a straight arrow bulb is present, it controls the straight lane.

[0061] Referring now to FIG4, FIG4 illustrates another example scenario in which the association of traffic control devices with lanes can be determined according to some embodiments of the present disclosure. Environment 400 may include a plurality of traffic control devices 402(1)-402(4) (hereinafter collectively referred to as “traffic control devices 402”) and a plurality of lanes 404(1)-404(4) (hereinafter collectively referred to as “lanes 404”). In this scenario, one or more systems of the present disclosure may associate one or more of the traffic control devices 402 with one or more of the lanes 404. In the example of FIG4, the traffic control devices 402 correspond to traffic lights or traffic signals, and in some cases, one or more systems of the present disclosure may associate one or more active bulbs of the traffic control devices 402 with one or more of the lanes 404. For example, lane features corresponding to lane 404 and traffic control device features corresponding to traffic control device 402 can be applied to a deep neural network. The deep neural network can use these features to calculate confidence scores associated with possible pairings of traffic control device 402 and lane 404, and based on the confidence scores, one or more traffic control devices 402 can be associated with one or more lanes 404. For example, in the example of Figure 4, a first traffic control device 402 (1) can be associated with a first lane 404 (1), a second traffic control device 402 (2) can be associated with a second lane 404 (2), a third traffic control device 402 (3) can be associated with a third lane 404 (3), and a fourth traffic control device 402 (4) can be associated with a fourth lane 404 (4).

[0062] For example, Figure 5 is a data flow diagram illustrating an example of a process 300 of a machine learning model according to some embodiments of the present disclosure performing traffic control signal and lane association in a scenario similar to that shown in Figure 4. As illustrated in the example of Figure 5, one or more embedding layers 502 may use one or more traffic control device features 118 to generate one or more traffic control device embeddings 504. One or more traffic control device embeddings 504 may be fed into the backbone and classification head 506 of one or more machine learning models 108, and the backbone and classification head 506 may use one or more traffic control device embeddings 504 to compute one or more confidence scores 120 indicating the association between one or more traffic control devices and lanes.

[0063] Referring back to the example in Figure 4, in some examples, one or more systems can automatically and synthetically generate ground truth data indicating the association between traffic control equipment and lanes without using image information. To perform this operation for scenarios similar to those shown in the example of Figure 4 (e.g., involving a more simplified scenario), one or more systems can automatically and synthetically generate ground truth traffic light-lane associations without using image information. For this purpose, one or more systems can generate a 3D lane geometry based on traffic rules (e.g., specifying an effective lane width range) and given a random seed for the number of lanes. In some cases, one or more systems can project the lane 3D geometry onto a 2D image space using the camera's intrinsic and extrinsic parameters to obtain a 2D geometry. For each lane segment, one or more systems can generate the lane direction based on traffic rules (e.g., in a right-hand driving country, the right-turn lane should be the rightmost lane). Additionally, one or more systems can generate a 3D traffic light geometry based on traffic rules (e.g., specifying the effective traffic light placement and traffic light size on the lanes) and given a random seed for the number of lights. Using the camera's intrinsic and extrinsic parameters, one or more systems can project the 3D geometry of a traffic light onto a 2D image space to obtain a 2D geometry. One or more systems can generate target road users and states, represented as one of red, yellow, and green circles. Then, for each activated traffic light bulb, one or more systems can assign its associated lane segment based on the traffic light's proximity to the nearest lane segment in 3D.

[0064] Referring now to FIG6, FIG6 is a data flow diagram illustrating an example process 600 for training one or more machine learning models to determine associations between traffic control devices and corresponding lanes according to some embodiments of the present disclosure. As shown, one or more machine learning models 612 (which may correspond to one or more machine learning models 108) can be trained using input data 602 (e.g., training input). Input data 602 may include feature vectors (e.g., non-image feature vectors) similar to one or more lane features 116 and / or one or more traffic control device features 118 described herein. One or more machine learning models 612 can be trained using training input data 602 and corresponding ground truth data 604 (which may correspond to input data 602). In some examples, ground truth data 604 may include various data indicating valid associations between traffic control devices and lanes, including annotations, labels, masks, values ​​(e.g., confidence values), etc. For example, in some embodiments, ground truth data 604 may indicate actual confidence values ​​associated with potential pairings of traffic control devices and lanes from input data 602.

[0065] Input data 602 and ground truth data 604 can be included as part of a training dataset generated by dataset generator 616 based at least on rule data 618 (e.g., synthetically generated). For example, input data 602 can be synthesized based on traffic rules governing traffic light placement at intersections and other scenarios, while ground truth data 604 can be generated based on the proximity of traffic lights to lanes and / or on rules defining various traffic regulations (e.g., left-arrow bulbs control left-turn lanes, right-arrow bulbs control right-turn lanes, solid-circle bulbs control regular lanes, straight-arrow bulbs control straight lanes, etc.), where image data may not be required. Additionally, in any example, ground truth data 604 can be synthetically generated (e.g., generated from computer models or renderings), realistically generated (e.g., designed and generated from real-world data), machine-automated (e.g., extracting features from data using feature analysis and learning and then generating labels), human-annotated (e.g., labelers or annotation experts defining label placement), and / or a combination thereof (e.g., human identification of polyline vertices, machine generation of polygons using a polygon rasterizer).

[0066] Training engine 608 may use one or more loss functions that measure the loss (e.g., error) when comparing output data 610 generated by one or more machine learning models 612 with ground truth data 604 and / or input data 602. In some examples, training engine 608 may compare output data 610 from one or more machine learning models 612 with input data 602 and update 614 one or more parameters 606 of one or more machine learning models 612 based at least on that comparison. That is, training engine 608 may update / optimize one or more parameters 606 associated with one or more machine learning models 612 to reduce the loss / difference between output data 610 (e.g., predicted association and / or confidence of traffic control equipment with lanes) and ground truth data 604 (e.g., ground truth association and / or confidence of traffic control equipment with lanes). Any type of loss function may be used, such as cross-entropy loss, mean squared error, mean absolute error, mean bias error, and / or other loss function types. In some examples, different outputs may have different loss functions. In such an example, loss functions can be combined to form a total loss, and this total loss can be used to train (e.g., update its parameters) one or more machine learning models 612. In any example, backpropagation computation can be performed to recursively compute the gradient of the loss function with respect to the training parameters. In some examples, the weights and biases of one or more machine learning models 612 can be used to compute these gradients.

[0067] In some examples, one or more machine learning models 612 may be packaged as microservices (such as inference microservices (e.g., NVIDIA NIM)), which may include containers (e.g., operating system (OS) level virtualization packages) that may include an application programming interface (API) layer, a server layer, a runtime layer, and / or a model "engine". For example, an inference microservice may include the container itself and the model (e.g., weights and biases). In some cases, such as when one or more machine learning models 612 are small enough (e.g., have a sufficiently small number of parameters), one or more models 612 may be contained within the container itself. In some embodiments, one or more machine learning models 612 described herein may be deployed as inference microservices to accelerate model deployment on any cloud, data center, or edge computing system while ensuring data security. For example, an inference microservice may include one or more APIs, pre-configured containers for simplified deployment, an optimized inference engine (e.g., execution software built using standardized AI model deployments, such as NVIDIA's Triton Inference Server, and / or one or more APIs for high-performance deep learning inference, which may include inference runtimes and model optimizations that provide low latency and high throughput for production applications, such as NVIDIA's TensorRT), and / or enterprise management data for telemetry (e.g., including identity, metrics, health checks, and / or monitoring). One or more machine learning models 612 described herein, along with acceleration infrastructure, may be included as part of the microservice, capable of deployment using a single command and / or orchestration and auto-scaling using a container orchestration system on the acceleration infrastructure (e.g., up to data center scale on a single device). Therefore, an inference microservice may include one or more machine learning models 612 (e.g., already optimized for high-performance inference), inference runtime software for executing one or more machine learning models 612 and providing output / response to inputs (e.g., user queries, prompts, etc.), and enterprise management software for providing health checks, identity, and / or other monitoring. In some embodiments, the inference microservice may include software for performing in-situ replacements and / or updates to one or more machine learning models 612. When a replacement or update is performed, the software performing the replacement / update may maintain user configurations for the inference runtime software and the enterprise management software.

[0068] Figure 7 illustrates an example of a system 702 that can perform one or more processes described herein, according to some embodiments of the present disclosure. As shown, system 702 (which may represent and / or include example computing device 1200 and / or example data center 1300) may include one or more processors 704 (which may resemble and / or include CPU 1206 and / or GPU 1208) and memory 706 (which may resemble and / or include memory 1204). For example, memory 706 may store one or more of lane sensor 104, traffic control device sensor 106, one or more machine learning models 108 and / or associated components 110. Additionally, one or more processors 704 may execute one or more of lane sensor 104, traffic control device sensor 106, one or more machine learning models 108 and / or associated components 110 to perform one or more processes described herein.

[0069] For example, system 702 can receive sensor data 114 from one or more sensors 102 of machine 708 (which may correspond to autonomous vehicle 1100). Sensor data 114 can be processed by lane sensor 104 and traffic control device sensor 106, which can output non-image features associated with lanes and traffic control devices in the environment. These non-image features can be applied to one or more machine learning models 108, which can calculate confidence scores associated with potential traffic control device-lane pairings. Association component 110 can use these confidence scores to associate one or more traffic control devices with one or more corresponding lane segments. These associations 122 between one or more traffic control devices and lanes can be sent to one or more driving stack components 112 of machine 708. One or more driving stack components 112 can enable machine 708 to perform one or more control operations based on the associations 122 between one or more traffic control devices and lanes.

[0070] Referring now to Figures 8 through 10, each block of the methods 800, 900, and 1000 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. Methods 800, 900, and 1000 can also be embodied as computer-usable instructions stored on a computer storage medium. Methods 800, 900, and 1000 can be provided by a standalone application, service, or managed service (standalone or in combination with another managed service) or plug-in to another product, to name a few. Furthermore, methods 800, 900, and 1000 are described by way of example with respect to Figure 1. However, methods 800, 900, and 1000 can be additionally or alternatively performed by any one or any combination of systems, including but not limited to the systems described herein.

[0071] Figure 8 is a flowchart illustrating an example of a method 800 for associating an activated traffic control signal with a lane segment according to some embodiments of the present disclosure. Method 800 at block B802 may include applying input data to one or more machine learning models, the input data including at least first feature data representing at least one of first geometric information or first semantic information corresponding to a plurality of lane segments, and second feature data representing at least one of second geometric information or second semantic information corresponding to a plurality of traffic control signals. For example, one or more lane features 116 and one or more traffic control device features 118 may be applied to one or more machine learning models 108. In some examples, one or more lane features 116 may include one or more feature vectors indicating lane geometry and / or lane semantics (e.g., lane direction). In some cases, one or more traffic control device features 118 may include one or more feature vectors indicating geometry (e.g., shape, position, etc.) and / or semantics (e.g., expected road users, state, etc.) associated with a traffic control device (e.g., activated traffic light).

[0072] Method 800 at box B804 may include using one or more machine learning models and calculating multiple confidence scores based at least on input data, the confidence scores indicating whether one or more activation signals of multiple traffic control signals correspond to one or more lane segments of multiple lane segments. For example, one or more machine learning models 108 may calculate one or more confidence scores 120 based at least on one or more lane features 116 and / or one or more traffic control device features 118.

[0073] Method 800 at box B806 may include associating one or more activation signals with one or more lane segments based on at least a plurality of confidence scores. For example, association component 110 may associate one or more activation signals with one or more lane segments based on at least one or more confidence scores 120 calculated by one or more machine learning models 108. In some cases, association component 110 may analyze one or more confidence scores 120 and / or other data (e.g., geometric or 3D proximity) to determine whether to associate a traffic control device with a lane. For example, association component 110 may determine the highest confidence score for each lane-to-traffic control device pairing and then select the pairing with the highest score as a valid pairing. In some cases, association component 110 may determine whether the pairing with the highest score is above a threshold compared to pairs with lower scores before identifying a pairing as a valid pairing. For example, if two potential pairs of lights and lanes seem to be associated based on confidence scores with similar values, association component 110 may output uncertainty about the pairing or not output a valid pairing at all.

[0074] Method 800 at block B808 may include causing the machine to perform one or more operations based at least on associations. For example, one or more driving stack components 112 may cause the machine to perform one or more operations based at least on associations 122 between one or more traffic control devices and lanes. In various examples, one or more operations may include, but are not limited to, stopping the machine, changing lanes, maintaining the machine's current trajectory, updating the path, etc.

[0075] Referring now to FIG9, FIG9 is a flowchart illustrating an example of a method 900 for associating traffic control devices with lanes according to some embodiments of the present disclosure. Method 900, at block B902, may include applying at least one or more first features corresponding to a plurality of lanes and one or more second features corresponding to a plurality of traffic control devices to one or more machine learning models. For example, one or more lane features 116 and one or more traffic control device features 118 may be applied to one or more machine learning models 108. In some examples, one or more lane features 116 may include one or more feature vectors indicating lane geometry and / or lane semantics (e.g., lane direction). In some cases, one or more traffic control device features 118 may include one or more feature vectors indicating geometry (e.g., shape, position, etc.) and / or semantics (e.g., expected road users, state, etc.) associated with a traffic control device (e.g., activating traffic lights).

[0076] Method 900 at box B904 may include associating at least one first traffic control device among a plurality of traffic control devices with at least one first lane among a plurality of lanes based on at least one or more machine learning models that process one or more first features and one or more second features. For example, association component 110 may determine association 122 of one or more traffic control devices with lanes based on at least one or more confidence scores 120 calculated by one or more machine learning models 108, which may include associating a first traffic control device with a first lane segment. In some cases, association component 110 may analyze one or more confidence scores 120 and / or other data (e.g., geometric or 3D proximity) to determine whether to associate a traffic control device with a lane. For example, association component 110 may determine the highest confidence score for each lane-to-traffic control device pairing and then select the pairing with the highest score as a valid pairing. In some cases, association component 110 may determine whether the pairing with the highest score is above a threshold compared to pairs with lower scores before identifying a pairing as a valid pairing. For example, if two potential light-lane association pairs seem possible based on confidence scores with similar values, the association component 110 may output uncertainty about the pairing, or not output a valid pairing at all.

[0077] Method 900 at block B906 may include causing the machine to perform one or more control operations based on at least one association. For example, one or more driving stack components 112 may cause the machine to perform one or more operations based on at least one or more traffic control device associations 122 with lanes. In various examples, one or more operations may include, but are not limited to, stopping the machine, changing lanes, maintaining the machine's current trajectory, updating the path, etc.

[0078] Figure 10 is a flowchart illustrating an example of a method 1000 for synthetically generating data for training a machine learning model to determine the association between traffic control devices and lanes, according to some embodiments of the present disclosure. Method 1000 may include generating a 3D lane geometry at block B1002. For example, a dataset generator 616 may generate the 3D lane geometry. In some examples, the dataset generator 616 may generate the 3D lane geometry based at least on rule data 618, which may represent traffic rules. Traffic rules may indicate a range of effective lane widths. In some examples, a random number of lanes may be generated.

[0079] Method 1000 at box B1004 may include projecting the 3D lane geometry onto a 2D image space to obtain a 2D lane geometry. For example, dataset generator 616 may project the 3D lane geometry onto a 2D image space to obtain a 2D lane geometry. In some examples, the 3D lane geometry may be projected onto a 2D image space using either intrinsic or extrinsic parameters of the camera. Method 1000 at box B1006 may include generating one or more lane directions for each lane segment. For example, dataset generator 616 may generate lane directions for each lane segment. In some examples, dataset generator 616 may generate lane directions for each lane segment based on traffic rules in rule data 618 (e.g., in right-hand drive driving, the right-turn lane should be the rightmost lane).

[0080] Method 1000 at box B1008 may include generating a 3D traffic light geometry. For example, a dataset generator 616 may generate a 3D traffic light geometry. In some examples, the dataset generator 616 may generate the 3D traffic light geometry based on traffic rules in rule data 618 (e.g., which specifies the effective placement of traffic lights on lanes and traffic light sizes) and given a random seed for the number of lights. Method 1000 at box B1010 may include projecting the 3D traffic light geometry into a 2D image space to obtain a 2D traffic light geometry. For example, the dataset generator 616 may project the 3D traffic light geometry into a 2D image space to obtain a 2D traffic light geometry. In some examples, the 3D traffic light geometry may be projected into a 2D image space using intrinsic and / or extrinsic parameters of the camera.

[0081] Method 1000 at box B1012 may include generating target road users and states for each traffic light. For example, dataset generator 616 may generate target road users and states for each traffic light. In some examples, dataset generator 616 may generate target road users and states for each traffic light based on traffic rules. In some cases, for a traffic light bulb with multiple arrows (e.g., left arrow and straight arrow), its state may be represented as multiple arrow states.

[0082] Method 1000 at box B1014 may include assigning each active traffic light bulb to its associated lane segment based at least on at least one of traffic rules or proximity. For example, dataset generator 616 may assign each traffic light bulb to its associated lane segment based at least on traffic rules or the proximity of the traffic light bulb to the lane segment. For example, bulbs may be paired with lane segments based on a priority order of following traffic rules. Such priority order may include, but is not limited to (e.g., other combinations or orders are also possible), (1) lights for road users other than vehicles are not associated with any lane segment; (2) lights are not facing their own road (from the direction) or are not at the current intersection (from the 3D geometry); (3) if a U-turn arrow bulb is present, it controls only the U-turn lane; (4) if a left arrow bulb is present, it controls the left-turn lane; if a U-turn bulb is not present, it also controls the left U-turn lane; (5) if a right arrow bulb is present, it controls the right-turn lane; (6) if a solid circle bulb is present, it controls the remaining lanes; (7) if a straight arrow bulb is present, it controls the straight lane. Additionally or alternatively, for each activated traffic light bulb, one or more lane segments may be assigned to it based on how close the traffic light is to its nearest lane segment in 3D.

[0083] The systems and methods described herein can be used for a variety of purposes, such as, but not limited to, machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, traffic control devices and lane association, data center processing, conversational AI, optical transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation of 3D assets, cloud computing, generative AI, (large) language models and / or any other suitable applications.

[0084] The disclosed embodiments may be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines), systems implemented using robots, aviation systems, medical systems, boating systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for associating traffic control equipment with lane segment directions, systems implemented using edge devices, systems containing one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in a data center, systems for performing conversational AI operations, systems for performing optical transmission simulations, systems for performing collaborative content creation of 3D assets, systems implemented at least partially using cloud computing resources, systems for performing generative AI operations, systems that implement or perform operations using large language models (LLMs), and / or other types of systems.

[0085] Example autonomous vehicles

[0086] Figure 11A is an illustration of an example autonomous vehicle 1100 according to some embodiments of the present disclosure. The autonomous vehicle 1100 (or, alternatively, referred to herein as “vehicle 1100”) may include, but is not limited to, passenger vehicles such as cars, trucks, buses, first-response vehicles, shuttles, electric or motorized bicycles, motorcycles, fire trucks, police vehicles, ambulances, boats, engineering vehicles, submarines, robotic vehicles, drones, aircraft, vehicles coupled to trailers (e.g., semi-trailer trucks for hauling goods) and / or other types of vehicles (e.g., driverless and / or vehicles accommodating one or more passengers). Autonomous vehicles are typically described according to the levels of automation defined by the National Highway Traffic Safety Administration (NHTSA), a division of the U.S. Department of Transportation, and the Society of Automotive Engineers (SAE) in its "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles" (Standard No. J3016-201806, published June 15, 2018; Standard No. J3016-201609, published September 30, 2016; and previous and future versions of this standard). Vehicle 1100 may be able to implement one or more functions that meet Level 3-5 of the autonomous driving level. Vehicle 1100 may be able to implement one or more functions that meet Level 1-5 of the autonomous driving level. For example, depending on the embodiment, vehicle 1100 may be able to implement driver assistance (Level 1), partial automation (Level 2), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5). As used herein, the term “autonomy” can include any and / or all types of autonomy of the vehicle 1100 or other machine, such as full autonomy, high autonomy, conditional autonomy, partial autonomy, assisted autonomy, semi-autonomy, primary autonomy or other designations.

[0087] Vehicle 1100 may include components such as chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other vehicle components. Vehicle 1100 may include a propulsion system 1150, such as an internal combustion engine, a hybrid power plant, an all-electric motor, and / or another type of propulsion system. Propulsion system 1150 may be connected to the drivetrain of vehicle 1100, which may include a transmission, to enable propulsion of vehicle 1100. Propulsion system 1150 may be controlled in response to receiving a signal from throttle / accelerator 1152.

[0088] A steering system 1154, which may include a steering wheel, can be used to steer the vehicle 1100 (e.g., along a desired path or route) when the propulsion system 1150 is operating (e.g., when the vehicle is in motion). The steering system 1154 may receive signals from the steering actuator 1156. For fully automatic (level 5) functionality, the steering wheel may be optional.

[0089] The brake sensor system 1146 can be used to operate the vehicle brakes in response to receiving signals from the brake actuator 1148 and / or the brake sensor.

[0090] One or more controllers 1136, which may include one or more system-on-a-chip (SoC) 1104 (FIG. 11C) and / or one or more GPUs, may provide signals (e.g., signals representing commands) to one or more components and / or systems of vehicle 1100. For example, one or more controllers may send signals to operate vehicle brakes via one or more brake actuators 1148, to operate steering system 1154 via one or more steering actuators 1156, and to operate propulsion system 1150 via one or more throttles / accelerators 1152. One or more controllers 1136 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operating commands (e.g., signals representing commands) to enable autonomous driving and / or assist a human driver in driving vehicle 1100. One or more controllers 1136 may include a first controller 1136 for autonomous driving functions, a second controller 1136 for functional safety functions, a third controller 1136 for artificial intelligence functions (e.g., computer vision), a fourth controller 1136 for infotainment functions, a fifth controller 1136 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 1136 may handle two or more of the above functions, two or more controllers 1136 may handle a single function, and / or any combination thereof.

[0091] One or more controllers 1136 may provide signals for controlling one or more components and / or systems of vehicle 1100 in response to sensor data (e.g., sensor inputs) received from one or more sensors. Sensor data may be received from, for example, but not limited to, Global Navigation Satellite System (“GNSS”) sensors 1158 (e.g., Global Positioning System sensors), RADAR sensors 1160, ultrasonic sensors 1162, LIDAR sensors 1164, inertial measurement unit (IMU) sensors 1166 (e.g., accelerometers, gyroscopes, magnetic compasses, magnetometers, etc.), microphones 1196, stereo cameras 1168, wide-angle cameras 1170 (e.g., fisheye cameras), infrared cameras 1172, surround cameras 1174 (e.g., 360-degree cameras), long-range and / or medium-range cameras 1198, speed sensors 1144 (e.g., for measuring the rate of vehicle 1100), vibration sensors 1142, steering sensors 1140, braking sensors (e.g., as part of braking sensor system 1146), and / or other sensor types.

[0092] One or more of the controllers 1136 may receive input (e.g., represented by input data) from the instrument cluster 1132 of the vehicle 1100 and provide output (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display 1134, an auditory signaling device, a speaker, and / or via other components of the vehicle 1100. These outputs may include information such as vehicle speed, rate, time, map data (e.g., a high-definition (“HD”) map 1122 of Figure 11C), location data (e.g., the location of the vehicle 1100 on a map), direction, the location of other vehicles (e.g., grid occupancy), and information about objects and their states perceived by the controllers 1136. For example, the HMI display 1134 may display information about the presence of one or more objects (e.g., street signs, warning signs, traffic light changes, etc.) and / or information about driving maneuvers that the vehicle has made, is making, or will make (e.g., changing lanes now, leaving 34B in two miles, etc.).

[0093] Vehicle 1100 also includes a network interface 1124, which can communicate via one or more networks using one or more wireless antennas 1126 and / or a modem. For example, network interface 1124 may be able to communicate via Long Term Evolution (“LTE”), Wideband Code Division Multiple Access (“WCDMA”), Universal Mobile Telecommunications System (“UMTS”), Global System for Mobile Communications (“GSM”), IMT-CDMA Multicarrier (“CDMA2000”), etc. One or more wireless antennas 1126 may also enable communication between objects in the environment (e.g., vehicles, mobile devices, etc.) using one or more local area networks such as Bluetooth, Bluetooth Low Energy (“LE”), Z-Wave, ZigBee, etc., and / or one or more low-power wide area networks (LPWANs such as LoRaWAN, SigFox, etc.).

[0094] Figure 11B is an example of camera positions and fields of view for the example autonomous vehicle 1100 of Figure 11A according to some embodiments of the present disclosure. The cameras and their respective fields of view are an example embodiment and are not intended to be limiting. For example, additional and / or replaceable cameras may be included, and / or these cameras may be located at different positions on the vehicle 1100.

[0095] The camera type used for the camera may include, but is not limited to, a digital camera suitable for use with components and / or systems of vehicle 1100. The camera may operate at Automotive Safety Integrity Level (ASIL) B and / or another ASIL. The camera type may have any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc., depending on the embodiment. The camera may be able to use a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include a red-white-white-white (RCCC) color filter array, a red-white-white-blue (RCCB) color filter array, a red-blue-green-white (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, a sharp-pixel camera, such as a camera with RCCC, RCCB, and / or RBGC color filter arrays, may be used in efforts to improve light sensitivity.

[0096] In some examples, one or more of the cameras can be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function monocular camera can be installed to provide functions including lane departure warning, traffic sign assistance, and intelligent headlight control. One or more of the cameras (e.g., all cameras) can simultaneously record and provide image data (e.g., video).

[0097] One or more of the cameras can be mounted in mounting components such as custom-designed (3D-printed) components to cut off stray light and reflections from inside the vehicle (e.g., reflections from the dashboard reflected in the windshield mirror) that may interfere with the camera's image data capture capabilities. Regarding the wing mirror mounting components, the wing mirror components can be custom-3D printed so that the camera mounting plate matches the shape of the wing mirror. In some examples, one or more cameras can be integrated into the wing mirror. For side-view cameras, one or more cameras can also be integrated into the four pillars at each corner of the cab.

[0098] A camera with a field of view that includes the environment in front of the vehicle 1100 (e.g., a front-facing camera) can be used for surround view to help identify forward paths and obstacles, and, with the assistance of one or more controllers 1136 and / or control SoCs, to provide information crucial for generating an occupancy grid and / or determining a preferred vehicle path. The front-facing camera can be used to perform many of the same ADAS functions as LiDAR, including emergency braking, pedestrian detection, and collision avoidance. The front-facing camera can also be used in ADAS functions and systems, including Lane Departure Warning (“LDW”), Autonomous Cruise Control (“ACC”), and / or other functions such as traffic sign recognition.

[0099] A variety of cameras can be used in the front-end configuration, including, for example, monocular camera platforms including complementary metal-oxide-semiconductor (“CMOS”) color imagers. Another example could be a wide-angle camera 1170, which can be used to perceive objects entering the field of view from the periphery (e.g., pedestrians, intersection traffic, or bicycles). Although only one wide-angle camera is illustrated in Figure 11B, any number (including zero) of wide-angle cameras 1170 can be present on vehicle 1100. Furthermore, any number of one or more remote cameras 1198 (e.g., long-view stereo camera pairs) can be used for depth-based object detection, especially for objects for which neural networks have not yet been trained. One or more remote cameras 1198 can also be used for object detection and classification, as well as basic object tracking.

[0100] Any number of stereo cameras 1168 may also be included in a front-mounted configuration. In at least one embodiment, one or more stereo cameras 1168 may include an integrated control unit comprising a scalable processing unit that can provide a multi-core microprocessor and programmable logic (“FPGA”) with an integrated controller area network (“CAN”) or Ethernet interface on a single chip. Such a unit can be used to generate a 3D map of the vehicle environment, including distance estimates for all points in the image. Alternative stereo cameras 1168 may include a compact stereo vision sensor that may include two camera lenses (one on each side) and an image processing chip capable of measuring the distance from the vehicle to a target object and using the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 1168 may be used in addition to those described herein or alternatively.

[0101] Cameras (e.g., side-view cameras) having a field of view that includes the sides of the vehicle 1100 can be used for surround view, providing information for creating and updating occupancy grids and generating side-impact collision warnings. For example, surround cameras 1174 (e.g., four surround cameras 1174 as shown in Figure 11B) can be positioned on the vehicle 1100. Surround cameras 1174 can include wide-angle cameras 1170, fisheye cameras, 360-degree cameras, and / or the like. Four examples: four fisheye cameras can be positioned at the front, rear, and sides of the vehicle. In an alternative arrangement, the vehicle can use three surround cameras 1174 (e.g., left, right, and rear) and can utilize one or more other cameras (e.g., forward-facing cameras) as a fourth surround-view camera.

[0102] A camera (e.g., a rear-view camera) having a field of view that includes the environment behind the vehicle 1100 can be used for parking assistance, surround view, rear collision warning, and creating and updating occupancy grids. A wide variety of cameras can be used, including but not limited to those also suitable as front-facing cameras as described herein (e.g., long-range and / or mid-range camera 1198, stereo camera 1168, infrared camera 1172, etc.).

[0103] Figure 11C is a block diagram of an example system architecture for the example autonomous vehicle 1100 of Figure 11A according to some embodiments of the present disclosure. It should be understood that this arrangement and other arrangements described herein are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in combination with other components, and in any suitable combination and location. The various functions described herein as being performed by these entities can be implemented by hardware, firmware, and / or software. For example, the various functions can be implemented by a processor executing instructions stored in memory.

[0104] In Figure 11C, each component, feature, and system of vehicle 1100 is illustrated as being connected via bus 1102. Bus 1102 may include a Controller Area Network (CAN) data interface (or, alternatively, referred to herein as the "CAN bus"). CAN may be a network within vehicle 1100 used to assist in the control of various features and functions of vehicle 1100, such as the actuation of brakes, acceleration, braking, steering, windshield wipers, etc. The CAN bus may be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., CAN ID). The CAN bus can be read to find steering wheel angle, ground speed, engine speed per minute (RPM), button positions, and / or other vehicle status indicators. The CAN bus may be ASIL B compliant.

[0105] Although bus 1102 is described herein as a CAN bus, this is not intended to be limiting. For example, FlexRay and / or Ethernet may be used in addition to or alternatively to a CAN bus. Furthermore, although bus 1102 is represented by a single line, this is not intended to be limiting. For example, any number of buses 1102 may exist, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 1102 may be used to perform different functions and / or may be used for redundancy. For example, a first bus 1102 may be used for a collision avoidance function, and a second bus 1102 may be used for drive control. In any example, each bus 1102 may communicate with any component of vehicle 1100, and two or more buses 1102 may communicate with the same component. In some examples, each SoC 1104, each controller 1136, and / or each computer within the vehicle may have access to the same input data (e.g., input from sensors in the vehicle 1100) and may be connected to a common bus such as the CAN bus.

[0106] Vehicle 1100 may include one or more controllers 1136, such as those described herein with respect to Figure 11A. Controller 1136 can be used for a wide variety of functions. Controller 1136 can be coupled to any other different components and systems of vehicle 1100 and can be used for the control of vehicle 1100, artificial intelligence of vehicle 1100, infotainment and / or the like for vehicle 1100.

[0107] Vehicle 1100 may include one or more System-on-Chip (SoC) 1104. SoC 1104 may include CPU 1106, GPU 1108, processor 1110, cache 1112, accelerator 1114, data storage 1116, and / or other components and features not shown. SoC 1104 can be used to control vehicle 1100 across a wide variety of platforms and systems. For example, one or more SoCs 1104 may be integrated with an HD map 1122 in a system (e.g., the system of vehicle 1100), the HD map receiving map refreshes and / or updates from one or more servers (e.g., one or more servers 1178 in Figure 11D) via network interface 1124.

[0108] CPU 1106 may include CPU clusters or CPU complexes (or, alternatively, referred to herein as "CCPLEX"). CPU 1106 may include multiple cores and / or L2 cache. For example, in some embodiments, CPU 1106 may include eight cores in a coherent multiprocessor configuration. In some embodiments, CPU 1106 may include four dual-core clusters, each cluster having a dedicated L2 cache (e.g., 2MB L2 cache). CPU 1106 (e.g., CCPLEX) may be configured to support simultaneous cluster operation, such that any combination of clusters of CPU 1106 can be active at any given time.

[0109] CPU 1106 can implement power management capabilities including one or more of the following features: automatic clock gating of hardware blocks when idle to save dynamic power; clock gating of each core when the core is not actively executing instructions due to the execution of WFI / WFE instructions; independent power gating of each core; independent clock gating of each core cluster when all cores are clock-gated or power-gated; and / or independent power gating of each core cluster when all cores are power-gated. CPU 1106 can further implement enhanced algorithms for managing power states, wherein allowed power states and desired wake-up times are specified, and the hardware / microcode determines the optimal power state to enter for the core, cluster, and CCPLEX. The processing core can support simplified power state entry sequences in software, with this work offloaded to the microcode.

[0110] GPU 1108 may include an integrated GPU (or, alternatively, referred to herein as an "iGPU"). GPU 1108 may be programmable and efficient for parallel workloads. In some examples, GPU 1108 may use an enhanced tensor instruction set. GPU 1108 may include one or more streaming microprocessors, wherein each streaming microprocessor may include an L1 cache (e.g., an L1 cache with at least 96KB of storage capacity), and two or more of these streaming microprocessors may share an L2 cache (e.g., an L2 cache with 512KB of storage capacity). In some embodiments, GPU 1108 may include at least eight streaming microprocessors. GPU 1108 may use a computation application programming interface (API). Furthermore, GPU 1108 may use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0111] In automotive and embedded applications, the GPU 1108 can be power-optimized for optimal performance. For example, the GPU 1108 can be fabricated on FinFETs. However, this is not intended to be limiting, and the GPU 1108 can be fabricated using other semiconductor manufacturing processes. Each streaming microprocessor can combine several mixed-precision processing cores divided into multiple blocks. For example, and without limitation, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA Tensor cores for deep learning matrix arithmetic, an L0 instruction cache, a warp scheduler, a dispatch unit, and / or a 64KB register file. Furthermore, the streaming microprocessor can include independent parallel integer and floating-point data paths to leverage the mixture of computation and addressing computations to provide efficient execution of workloads. Streaming microprocessors may include independent thread scheduling capabilities to allow for finer-grained synchronization and cooperation between parallel threads. Streaming microprocessors may include combined L1 data caches and shared memory units to improve performance while simplifying programming.

[0112] The GPU 1108 may include, in some examples, a High Bandwidth Memory (HBM) and / or a 16GB HBM2 memory subsystem providing a peak memory bandwidth of approximately 900GB / s. In some examples, in addition to HBM memory or alternatively, Synchronous Graphics Random Access Memory (SGRAM), such as Generation 5 Graphics Double Data Rate Synchronous Random Access Memory (GDDR5), may be used.

[0113] The GPU 1108 may include unified memory technology, which includes access counters to allow memory pages to be migrated more precisely to the processors that access them most frequently, thereby improving the efficiency of shared memory ranges between processors. In some examples, Address Translation Service (ATS) support can be used to allow the GPU 1108 to directly access the CPU 1106 page tables. In such examples, when the GPU 1108 Memory Management Unit (MMU) experiences a miss, the address translation request can be transferred to the CPU 1106. In response, the CPU 1106 can look up the virtual-physical mapping for the address in its page tables and transfer the translation back to the GPU 1108. Thus, unified memory technology can allow a single unified virtual address space for the memory of both the CPU 1106 and the GPU 1108, simplifying GPU 1108 programming and porting applications to the GPU 1108.

[0114] In addition, the GPU 1108 may include access counters that track how frequently the GPU 1108 accesses the memory of other processors. Access counters can help ensure that memory pages are moved to the physical memory of the processor that accesses those pages most frequently.

[0115] SoC 1104 may include any number of caches 1112, including those described herein. For example, cache 1112 may include an L3 cache available to both CPU 1106 and GPU 1108 (e.g., it is connected to both CPU 1106 and GPU 1108). Cache 1112 may include a write-back cache, which can track the state of rows, for example, using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). Depending on the embodiment, the L3 cache may include 4 MB or more, but a smaller cache size may also be used.

[0116] SoC 1104 may include an arithmetic logic unit (ALU) that can be utilized in processing of any of the various tasks or operations performed on vehicle 1100, such as processing a DNN. Furthermore, SoC 1104 may include a floating-point unit (FPU) (or other mathematical coprocessor or digital coprocessor type) for performing mathematical operations within the system. For example, SoC 1104 may include one or more FPUs integrated as execution units within CPU 1106 and / or GPU 1108.

[0117] SoC 1104 may include one or more accelerators 1114 (e.g., hardware accelerators, software accelerators, or combinations thereof). For example, SoC 1104 may include a hardware accelerator cluster, which may include optimized hardware accelerators and / or large on-chip memory. This large on-chip memory (e.g., 4MB SRAM) can enable the hardware accelerator cluster to accelerate neural networks and other computations. The hardware accelerator cluster can be used to complement GPU 1108 and offload some tasks from GPU 1108 (e.g., freeing up more cycles of GPU 1108 to perform other tasks). As an example, accelerator 1114 can be used for targeted workloads (e.g., perceptrons, convolutional neural networks (CNNs), etc.) that are sufficiently stable to allow for easy control of acceleration. When used herein, the term "CNN" can include all types of CNNs, including region-based or region convolutional neural networks (RCNNs) and fast RCNNs (e.g., for object detection).

[0118] Accelerator 1114 (e.g., a hardware accelerator cluster) may include a Deep Learning Accelerator (DLA). The DLA may include one or more Tensor Processing Units (TPUs) that can be configured to provide an additional 10 trillion operations per second for deep learning applications and inference. The TPU may be an accelerator configured to perform image processing functions (e.g., for CNNs, RCNNs, etc.) and optimized for performing image processing functions. The DLA may be further optimized for a specific set of neural network types and floating-point operations and inference. The DLA is designed to provide higher performance per millimeter than a general-purpose GPU and significantly outperform CPUs. The TPU can perform several functions, including single-instance convolution functions, support for INT8, INT16, and FP16 data types for both features and weights, and post-processor functions.

[0119] DLA can execute neural networks, especially CNNs, quickly and efficiently on processed or unprocessed data for any function across a wide variety of applications, such as, but not limited to: CNNs for object recognition and detection using data from camera sensors; CNNs for distance estimation using data from camera sensors; CNNs for emergency vehicle detection and recognition using data from microphones; CNNs for face recognition and vehicle owner recognition using data from camera sensors; and / or CNNs for safety and / or safety-related events.

[0120] The DLA can perform any function of the GPU 1108, and by using inference accelerators, for example, designers can make the DLA or GPU 1108 target any function. For example, designers can focus the CNN processing and floating-point operations on the DLA and leave other functions to the GPU 1108 and / or other accelerators 1114.

[0121] Accelerator 1114 (e.g., a cluster of hardware accelerators) may include a programmable vision accelerator (PVA), which may alternatively be referred to herein as a computer vision accelerator. The PVA may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. The PVA can provide a balance between performance and flexibility. For example, each PVA may include, for example, but not limited to, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors.

[0122] RISC cores can interact with image sensors (such as the image sensor of any camera described herein), image signal processors, and / or the like. Each of these RISC cores may include any amount of memory. Depending on the embodiment, the RISC core may use any of several protocols. In some examples, the RISC core may execute a real-time operating system (RTOS). RISC cores may be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, a RISC core may include an instruction cache and / or tightly coupled RAM.

[0123] DMA enables PVA components to access system memory independently of the CPU 1106. DMA can support any number of features to provide optimizations to the PVA, including but not limited to support for multidimensional addressing and / or circular addressing. In some examples, DMA can support addressing in up to six or more dimensions, which can include block width, block height, block depth, horizontal block step, vertical block step, and / or depth step.

[0124] A vector processor can be a programmable processor designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing capabilities. In some examples, a PVA may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem may operate as the main processing engine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or vector memory (e.g., VMEM). The VPU core may include a digital signal processor, such as, for example, a Single Instruction Multiple Data (SIMD) or Very Long Instruction Word (VLIW) digital signal processor. The combination of SIMD and VLIW can enhance throughput and speed.

[0125] Each of the vector processors may include an instruction cache and may be coupled to dedicated memory. As a result, in some examples, each of the vector processors may be configured to execute independently of other vector processors. In other examples, the vector processors included in a particular PVA may be configured to employ data parallelism. For example, in some embodiments, multiple vector processors included in a single PVA may execute the same computer vision algorithm, but on different regions of an image. In other examples, vector processors included in a particular PVA may execute different computer vision algorithms simultaneously on the same image, or even different algorithms on a sequence of images or portions of an image. Among other things, any number of PVAs may be included in a cluster of hardware accelerators, and any number of vector processors may be included in each of these PVAs. Furthermore, the PVA may include additional error correction code (ECC) memory to enhance overall system security.

[0126] Accelerator 1114 (e.g., a hardware accelerator cluster) may include an on-chip computer vision network and SRAM to provide high-bandwidth, low-latency SRAM for accelerator 1114. In some examples, on-chip memory may include at least 4MB of SRAM consisting of, for example, but not limited to, eight field-configurable memory blocks, accessible by both the PVA and DLA. Each pair of memory blocks may include an Advanced Peripheral Bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory may be used. The PVA and DLA may access memory via a backbone that provides high-speed memory access to the PVA and DLA. The backbone may include (e.g., using an APB) an on-chip computer vision network that interconnects the PVA and DLA to memory.

[0127] On-chip computer vision networks can include interfaces that ensure both the PVA and DLA provide ready and valid signals before transmitting any control signals / addresses / data. Such interfaces can provide separate phases and channels for transmitting control signals / addresses / data, as well as burst communication for continuous data transmission. This type of interface can conform to ISO 26262 or IEC 61508 standards, but other standards and protocols can also be used.

[0128] In some examples, SoC 1104 may include, for example, a real-time ray tracing hardware accelerator as described in U.S. Patent Application No. 16 / 101,232, filed August 10, 2018. This real-time ray tracing hardware accelerator can be used to quickly and efficiently determine the location and extent of objects (e.g., within a world model) to generate real-time visualization simulations for RADAR signal interpretation, sound propagation synthesis and / or analysis, SONAR system simulation, general wave propagation simulation, comparison with LiDAR data for localization and / or other functional purposes, and / or other uses. In some embodiments, one or more Tree Traversal Units (TTUs) may be used to perform one or more ray tracing-related operations.

[0129] Accelerators 1114 (e.g., hardware accelerator clusters) have broad applications in autonomous driving. PVAs can be programmable vision accelerators used in critical processing stages of ADAS and autonomous vehicles. The capabilities of PVAs are a good match for algorithmic domains requiring predictable processing, low power, and low latency. In other words, PVAs perform well in semi-dense or dense rule computation, even on small datasets requiring predictable runtimes with low latency and low power. Therefore, in the context of platforms for autonomous vehicles, PVAs are designed to run classical computer vision algorithms because they are effective in object detection and integer mathematical operations.

[0130] For example, according to one embodiment of this technology, PVA is used to perform computer stereo vision. In some examples, semi-global matching-based algorithms may be used, but this is not intended to be limiting. Many applications for Level 3-5 autonomous driving require instantaneous motion estimation / stereo matching (e.g., from moving structures, pedestrian recognition, lane detection, etc.). PVA can perform computer stereo vision functions on input from two monocular cameras.

[0131] In some examples, PVA can be used to perform intensive optical flow, providing processed RADAR data from the raw RADAR data (e.g., using 4D Fast Fourier Transform). In other examples, PVA is used for time-of-flight depth processing, which, for example, involves processing raw time-of-flight data to provide processed time-of-flight data.

[0132] DLA can be used to run any type of network to enhance control and driving safety, including, for example, neural networks that output a confidence metric for each object detection. Such a confidence value can be interpreted as a probability or as providing a relative “weight” for each detection compared to other detections. This confidence value allows the system to make further decisions about which detections should be considered true positives rather than false positives. For example, the system can set a threshold for the confidence and only consider detections exceeding the threshold as true positives. In an Automatic Emergency Braking (AEB) system, false positives can cause the vehicle to automatically perform emergency braking, which is clearly undesirable. Therefore, only the most confident detections should be considered as triggers for AEB. DLA can run neural networks to regress the confidence values. The neural network can take at least some subset of parameters as its input, such as bounding box dimensions, ground plane estimates obtained (e.g. from another subsystem), outputs from inertial measurement unit (IMU) sensors 1166 related to the orientation and distance of vehicle 1100, 3D position estimates of objects obtained from the neural network and / or other sensors (e.g., LiDAR sensor 1164 or RADAR sensor 1160), etc.

[0133] SoC 1104 may include one or more data storage units 1116 (e.g., memory). Data storage units 1116 may be on-chip memory of SoC 1104, which may store neural networks to be executed on the GPU and / or DLA. In some examples, for redundancy and security, data storage units 1116 may be large enough to store multiple instances of the neural network. Data storage units 1112 may include L2 or L3 cache 1112. References to data storage units 1116 may include references to memory associated with PVA, DLA, and / or other accelerators 1114 as described herein.

[0134] SoC 1104 may include one or more processors 1110 (e.g., embedded processors). Processor 1110 may include a startup and power management processor, which may be a dedicated processor and subsystem for handling startup power and management functions, as well as safety implementation. The startup and power management processor may be part of the SoC 1104 startup sequence and may provide runtime power management services. The startup power and management processor may provide clock and voltage programming, auxiliary system low-power state transitions, SoC 1104 thermal and temperature sensor management, and / or SoC 1104 power state management. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to the temperature, and SoC 1104 may use the ring oscillator to detect the temperature of CPU 1106, GPU 1108, and / or accelerator 1114. If it is determined that the temperature exceeds a threshold, the startup and power management processor may enter a temperature fault routine and place SoC 1104 into a lower power state and / or place vehicle 1100 into a driver-safe parking mode (e.g., safely stop vehicle 1100).

[0135] The processor 1110 may also include a set of embedded processors that can be used as an audio processing engine. The audio processing engine can be an audio subsystem that allows for full hardware support for multi-channel audio via multiple interfaces and a wide and flexible range of audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core with a digital signal processor and dedicated RAM.

[0136] The processor 1110 may also include an always-on-processor engine that can provide the necessary hardware features to support low-power sensor management and wake-up use cases. This always-on-processor engine may include a processor core, tightly coupled RAM, support for peripherals (such as timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0137] The processor 1110 may also include a security cluster engine, which includes a dedicated processor subsystem for handling security management for automotive applications. The security cluster engine may include two or more processor cores, tightly coupled RAM, support for peripheral devices (e.g., timers, interrupt controllers, etc.), and / or routing logic. In secure mode, the two or more cores may operate in lockstep mode and function as a single core with comparison logic that detects any differences between their operations.

[0138] The processor 1110 may also include a real-time camera engine, which may include a dedicated processor subsystem for handling real-time camera management.

[0139] The processor 1110 may also include a high dynamic range signal processor, which may include an image signal processor, which is a hardware engine that is part of the camera processing pipeline.

[0140] Processor 1110 may include a video image compositer, which may be (e.g., implemented on a microprocessor) a processing block, implementing video post-processing functions required by the video playback application to generate the final image for the player window. The video image compositer may perform lens distortion correction on the wide-angle camera 1170, the surround camera 1174, and / or the in-cabin monitoring camera sensor. The in-cabin monitoring camera sensor is preferably monitored by a neural network running on another instance of an advanced SoC, configured to recognize in-cabin events and respond accordingly. The in-cabin system may perform lip reading to activate mobile phone services and make calls, dictate emails, change vehicle destinations, activate or change the vehicle's infotainment system and settings, or provide voice-activated web browsing. Some functions are only available to the driver when the vehicle is operating in autonomous mode and are disabled in other situations.

[0141] Video image compositers can include enhanced temporal denoising for both spatial and temporal noise reduction. For example, in the case of motion in the video, denoising appropriately weights spatial information, reducing the weight of information provided by neighboring frames. In cases where the image or part of the image does not contain motion, the temporal denoising performed by the video image compositer can use information from previous images to reduce noise in the current image.

[0142] The video image compositer can also be configured to perform stereo correction on input stereo lens frames. When the operating system desktop is in use and the GPU 1108 does not need to continuously render new surfaces, the video image compositer can be further used for user interface components. Even when the GPU 1108 is powered on and active, performing 3D rendering, the video image compositer can be used to offload the GPU 1108 to improve performance and responsiveness.

[0143] The SoC 1104 may also include a Mobile Industry Processor Interface (MIPI) camera serial interface, a high-speed interface, and / or a video input block that can be used for camera and related pixel input functions for receiving video and input from a camera. The SoC 1104 may also include an input / output controller that can be software controlled and can be used to receive I / O signals not assigned to a specific role.

[0144] SoC 1104 may also include a wide range of peripheral interfaces to enable communication with peripherals, audio codecs, power management and / or other devices. SoC 1104 can be used to process data from cameras and sensors (e.g., LIDAR sensor 1164, RADAR sensor 1160, etc., which can be connected via Gigabit Multimedia Serial Link and Ethernet), data from bus 1102 (e.g., vehicle 1100 speed, steering wheel position, etc.), and data from GNSS sensor 1158 (connected via Ethernet or CAN bus). SoC 1104 may also include a dedicated high-performance, high-capacity memory controller, which may include its own DMA engine and can be used to free up CPU 1106 from routine data management tasks.

[0145] The SoC 1104 can be an end-to-end platform with a flexible architecture spanning Automation Levels 3-5, providing a comprehensive functional safety architecture that leverages and efficiently utilizes computer vision and ADAS technologies for diversity and redundancy, along with deep learning tools to deliver a flexible and reliable driving software stack. The SoC 1104 can be faster, more reliable, and even more energy- and space-efficient than conventional systems. For example, when combined with the CPU 1106, GPU 1108, and data storage 1116, the accelerator 1114 can provide a fast and efficient platform for Level 3-5 autonomous vehicles.

[0146] Therefore, this technology offers capabilities and functionalities that cannot be achieved through conventional systems. For example, computer vision algorithms can be executed on CPUs, which can be configured using high-level programming languages ​​such as C to execute a wide variety of processing algorithms across a diverse range of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. In particular, many CPUs cannot execute complex object detection algorithms in real time, which is a requirement for automotive ADAS applications and practical Level 3-5 autonomous vehicles.

[0147] In contrast to conventional systems, the techniques described in this paper, by providing CPU complexes, GPU complexes, and hardware accelerator clusters, allow multiple neural networks to be executed simultaneously and / or sequentially, and the results combined to achieve Level 3–5 autonomous driving capabilities. For example, a CNN executed on a DLA or dGPU (e.g., GPU 1120) could include text and word recognition, allowing a supercomputer to read and understand traffic signs, including those for which neural networks have not yet been specifically trained. The DLA could also include a neural network capable of recognizing, interpreting, and providing semantic understanding of the signs, and passing that semantic understanding to a path planning module running on the CPU complex.

[0148] As another example, multiple neural networks can operate simultaneously, as required for Level 3, 4, or 5 driving. For instance, a warning sign consisting of "Caution: Flashing lights indicate icy conditions," along with a light, can be interpreted independently or jointly by several neural networks. The sign itself can be recognized as a traffic sign by a deployed first neural network (e.g., a trained neural network), and the text "Flashing lights indicate icy conditions" can be interpreted by a deployed second neural network that informs the vehicle's path planning software (preferably executing on a CPU complex) that icy conditions exist when the flashing lights are detected. The flashing lights can be identified by a deployed third neural network operating across multiple frames, informing the vehicle's path planning software of the presence (or absence) of the flashing lights. All three neural networks can operate simultaneously, for example, within a DLA and / or on a GPU 1108.

[0149] In some examples, the CNN used for facial recognition and owner identification can use data from camera sensors to identify the presence of an authorized driver and / or owner of vehicle 1100. A processing engine always on the sensors can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and in safe mode, to disable the vehicle when the owner leaves. In this way, SoC 1104 provides security against theft and / or carjacking.

[0150] In another example, the CNN used for emergency vehicle detection and identification can use data from microphone 1196 to detect and identify emergency vehicle siren. In contrast to conventional systems that use a general classifier to detect siren and manually extract features, SoC 1104 uses a CNN to classify environmental and urban sounds as well as visual data. In a preferred embodiment, the CNN running on the DLA is trained to identify the relative shut-off rate of emergency vehicles (e.g., by using the Doppler effect). The CNN can also be trained to identify emergency vehicles specific to the localized area in which the vehicle operates, as identified by GNSS sensor 1158. Thus, for example, when operating in Europe, the CNN will seek to detect European siren, and when operating in the United States, the CNN will seek to identify siren only in North America. Once an emergency vehicle is detected, with the assistance of ultrasonic sensor 1162, the control program can be used to execute emergency vehicle safety routines, causing the vehicle to slow down, pull over to the side of the road, stop, and / or idle until the emergency vehicle passes.

[0151] The vehicle may include a CPU 1118 (e.g., a discrete CPU or dCPU) that can be coupled to the SoC 1104 via a high-speed interconnect (e.g., PCIe). The CPU 1118 may include, for example, an x86 processor. The CPU 1118 can be used to perform any of a wide variety of functions, including, for example, arbitrating the results of potential inconsistencies between ADAS sensors and the SoC 1104, and / or monitoring the status and health of the controller 1136 and / or the infotainment SoC 1130.

[0152] Vehicle 1100 may include a GPU 1120 (e.g., a discrete GPU or dGPU) that can be coupled to SoC 1104 via a high-speed interconnect (e.g., NVIDIA's NVLINK). GPU 1120 may provide additional artificial intelligence capabilities, for example by executing redundant and / or different neural networks, and can be used to train and / or update neural networks based at least in part on inputs (e.g., sensor data) from sensors of vehicle 1100.

[0153] Vehicle 1100 may also include a network interface 1124, which may include one or more wireless antennas 1126 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). Network interface 1124 can be used to enable wireless connectivity via the Internet to the cloud (e.g., with server 1178 and / or other network devices), with other vehicles, and / or with computing devices (e.g., passenger client devices). For communication with other vehicles, a direct link can be established between the two vehicles, and / or an indirect link can be established (e.g., across networks and via the Internet). A direct link can be provided using a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link can provide vehicle 1100 with information about vehicles approaching vehicle 1100 (e.g., vehicles in front, to the side, and / or behind vehicle 1100). This functionality may be part of vehicle 1100's cooperative adaptive cruise control function.

[0154] Network interface 1124 may include a SoC that provides modulation and demodulation functions and enables controller 1136 to communicate via a wireless network. Network interface 1124 may include an RF front-end for up-conversion from baseband to RF and down-conversion from RF to baseband. Frequency conversion can be performed using known processes and / or using a superheterodyne process. In some examples, the RF front-end functionality may be provided by a separate chip. The network interface may include wireless functions for communication via LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0155] Vehicle 1100 may also include data storage 1128, which may include off-chip (e.g., off-chip SoC 1104) storage devices. Data storage 1128 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash memory, hard disk, and / or other components and / or devices capable of storing at least one bit of data.

[0156] Vehicle 1100 may also include a GNSS sensor 1158. The GNSS sensor 1158 (e.g., GPS, assisted GPS sensor, differential GPS (DGPS) sensor, etc.) is used for auxiliary mapping, sensing, occupancy grid generation, and / or path planning functions. Any number of GNSS sensors 1158 can be used, including, for example, but not limited to, GPS using a USB connector with an Ethernet-to-serial (RS-232) bridge.

[0157] Vehicle 1100 may also include a RADAR sensor 1160. The RADAR sensor 1160 can be used by vehicle 1100 for remote vehicle detection even in dark and / or inclement weather conditions. The RADAR functional safety level may be ASIL B. The RADAR sensor 1160 can use CAN and / or bus 1102 (e.g., to transmit data generated by the RADAR sensor 1160) for control and access to object tracking data, and in some examples, Ethernet access for accessing raw data. A wide variety of RADAR sensor types can be used. For example, and without limitation, the RADAR sensor 1160 can be adapted for front, rear, and side RADAR use. In some examples, a pulse Doppler RADAR sensor is used.

[0158] RADAR sensor 1160 can include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range side coverage, etc. In some examples, long-range RADAR can be used for adaptive cruise control functions. A long-range RADAR system can provide a wide field of view (e.g., within 250m) achieved through two or more independent scans. RADAR sensor 1160 can help distinguish between stationary and moving objects and can be used by ADAS systems for emergency braking assist and forward collision warning. Long-range RADAR sensors can include a single-site multi-mode RADAR with multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In an example with six antennas, the four central antennas can create a focused beam pattern designed to record the vehicle 1100's surroundings at higher rates with minimal traffic interference from adjacent lanes. The other two antennas can extend the field of view, enabling rapid detection of vehicles entering or leaving the vehicle 1100's lane.

[0159] As an example, a mid-range RADAR system can include a range of up to 1160m (front) or 80m (rear) and a field of view of up to 42 degrees (front) or 1150 degrees (rear). Short-range RADAR systems can include, but are not limited to, RADAR sensors designed to be mounted at both ends of the rear bumper. When mounted at both ends of the rear bumper, such a RADAR sensor system can create two beams that continuously monitor blind spots behind and beside the vehicle.

[0160] Short-range RADAR systems can be used in ADAS systems for blind spot detection and / or lane change assistance.

[0161] Vehicle 1100 may also include ultrasonic sensors 1162. Ultrasonic sensors 1162, which may be positioned at the front, rear, and / or sides of vehicle 1100, can be used for parking assistance and / or creating and updating occupancy grids. A wide variety of ultrasonic sensors 1162 can be used, and different ultrasonic sensors 1162 can be used for different detection ranges (e.g., 2.5m, 4m). Ultrasonic sensors 1162 can operate at functional safety level ASIL B.

[0162] Vehicle 1100 may include a LIDAR sensor 1164. The LIDAR sensor 1164 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LIDAR sensor 1164 may be of functional safety level ASIL B. In some examples, vehicle 1100 may include multiple LIDAR sensors 1164 (e.g., two, four, six, etc.) that can use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0163] In some examples, the LiDAR sensor 1164 may be able to provide a list of objects and their distances within a 360-degree field of view. Commercially available LiDAR sensors 1164 may have an advertising range of, for example, approximately 1100m, with an accuracy of 2cm-3cm, and support for 1100Mbps Ethernet connectivity. In some examples, one or more non-protruding LiDAR sensors 1164 may be used. In such examples, the LiDAR sensor 1164 may be implemented as a small device that can be embedded in the front, rear, sides, and / or corners of the vehicle 1100. In such examples, the LiDAR sensor 1164 may provide a horizontal field of view of up to 120 degrees and a vertical field of view of 35 degrees, even for low-reflectivity objects, with a range of 200m. Front-mounted LiDAR sensors 1164 may be configured for a horizontal field of view between 45 and 135 degrees.

[0164] In some examples, LiDAR technologies such as 3D flash LiDAR can also be used. 3D flash LiDAR uses a flash of laser light as the emission source to illuminate the vehicle's surroundings up to approximately 200 meters. A flash LiDAR unit includes a receiver that records the laser pulse propagation time and reflected light on each pixel, which in turn corresponds to the range from the vehicle to the object. Flash LiDAR allows for the generation of highly accurate and distortion-free images of the surrounding environment using each laser flash. In some examples, four flash LiDAR sensors can be deployed, one on each side of the vehicle. Available 3D flash LiDAR systems include solid-state 3D staring array LiDAR cameras (e.g., non-scanning LiDAR devices) without moving parts other than a fan. Flash LiDAR devices can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture reflected laser light in the form of a 3D range point cloud and co-registered intensity data. By using a flash LIDAR, and because a flash LIDAR is a solid-state device with no moving parts, the LIDAR sensor 1164 is less susceptible to motion blur, vibration, and / or shock.

[0165] The vehicle may also include an IMU sensor 1166. In some examples, the IMU sensor 1166 may be located at the center of the rear axle of the vehicle 1100. The IMU sensor 1166 may include, for example, but not limited to, an accelerometer, a magnetometer, a gyroscope, a magnetic compass, and / or other sensor types. In some examples, such as in a six-axis application, the IMU sensor 1166 may include an accelerometer and a gyroscope, while in a nine-axis application, the IMU sensor 1166 may include an accelerometer, a gyroscope, and a magnetometer.

[0166] In some embodiments, the IMU sensor 1166 can be implemented as a miniature, high-performance GPS-assisted inertial navigation system (GPS / INS) that combines a microelectromechanical system (MEMS) inertial sensor, a high-sensitivity GPS receiver, and an advanced Kalman filter algorithm to provide estimates of position, velocity, and attitude. Thus, in some examples, the IMU sensor 1166 can enable the vehicle 1100 to estimate heading by directly observing and correlating velocity changes from GPS to the IMU sensor 1166 without requiring input from a magnetic sensor. In some examples, the IMU sensor 1166 and the GNSS sensor 1158 can be combined into a single integrated unit.

[0167] The vehicle may include a microphone 1196 placed in and / or around the vehicle 1100. Among other things, the microphone 1196 may be used for emergency vehicle detection and identification.

[0168] The vehicle may also include any number of camera types, including stereo camera 1168, wide-angle camera 1170, infrared camera 1172, surround camera 1174, long-range and / or mid-range camera 1198, and / or other camera types. These cameras can be used to capture image data around the entire perimeter of the vehicle 1100. The types of cameras used depend on the embodiment and the requirements of the vehicle 1100, and any combination of camera types can be used to provide the necessary coverage around the vehicle 1100. Furthermore, the number of cameras may vary depending on the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. As an example and without limitation, these cameras may support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet. Each of the cameras is described in more detail herein with reference to Figures 11A and 11B.

[0169] Vehicle 1100 may also include vibration sensor 1142. Vibration sensor 1142 can measure vibrations of vehicle components such as axles. For example, changes in vibration can indicate changes in the road surface. In another example, when two or more vibration sensors 1142 are used, differences between vibrations can be used to determine friction or slippage on the road surface (e.g., when there is a vibration difference between a power drive shaft and a free-rotating shaft).

[0170] Vehicle 1100 may include ADAS system 1138. In some examples, ADAS system 1138 may include SoC. ADAS system 1138 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keeping assist (LKA), blind spot warning (BSW), rear cross traffic warning (RCTW), collision warning system (CWS), lane centering (LC) and / or other features and functions.

[0171] The ACC system can use RADAR sensor 1160, LIDAR sensor 1164, and / or a camera. The ACC system may include longitudinal ACC and / or lateral ACC. Longitudinal ACC monitors and controls the distance to vehicles immediately in front of vehicle 1100 and automatically adjusts the vehicle speed to maintain a safe distance. Lateral ACC performs distance holding and, if necessary, advises vehicle 1100 to change lanes. Lateral ACC is associated with other ADAS applications such as LCA and CWS.

[0172] CACC uses information from other vehicles, which can be received indirectly from other vehicles via a wireless link or through a network connection (e.g., via the Internet) through network interface 1124 and / or wireless antenna 1126. Direct links can be provided by vehicle-to-vehicle (V2V) communication links, while indirect links can be infrastructure-to-vehicle (I2V) communication links. Typically, the V2V communication concept provides information about vehicles immediately ahead (e.g., vehicles immediately in front of vehicle 1100 and in the same lane), while the I2V communication concept provides information about traffic further ahead. A CACC system can include either or both of these I2V and V2V information sources. Given information about vehicles ahead of vehicle 1100, CACC can be more reliable, and it has the potential to improve traffic flow and reduce road congestion.

[0173] The Forward-Looking Warning (FCW) system is designed to alert the driver to hazards, enabling the driver to take corrective action. The FCW system uses a front-facing camera and / or RADAR sensor 1160 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating components. The FCW system can provide warnings in the form of, for example, audible, visual, haptic, and / or rapid braking pulses.

[0174] The AEB system detects an impending forward collision with another vehicle or other object and can automatically apply the brakes if the driver does not take corrective action within a specified time or distance parameter. The AEB system can use a front-facing camera and / or RADAR sensor 1160 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. When the AEB system detects a hazard, it typically first alerts the driver to take corrective action to avoid a collision, and if the driver does not take corrective action, the AEB system can automatically apply the brakes to attempt to prevent or at least mitigate the effects of the predicted collision. The AEB system may include technologies such as dynamic brake support and / or collision proximity braking.

[0175] The Lane Departure Warning (LDW) system provides visual, auditory, and / or tactile warnings, such as steering wheel or seat vibrations, to alert the driver when the vehicle crosses lane markings. When the driver indicates intentional lane departure, the LDW system is deactivated by activating a turn signal. The LDW system can utilize a front-facing camera coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating components.

[0176] The LKA system is a variant of the LDW system. If vehicle 1100 begins to leave the lane, the LKA system provides steering input or braking to correct vehicle 1100.

[0177] The BSW system detects and warns the driver of vehicles in the vehicle's blind spot. The BSW system can provide visual, auditory, and / or tactile alerts to indicate that merging or changing lanes is unsafe. The system can provide additional warnings when the driver uses turn signals. The BSW system can use a rear-facing camera and / or RADAR sensor 1160 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating components.

[0178] The RCTW system can provide visual, auditory, and / or tactile notifications when an object is detected outside the range of the rear camera while the vehicle 1100 is reversing. Some RCTW systems include AEB to ensure the application of the vehicle's brakes to avoid a collision. The RCTW system may use one or more rear-mounted RADAR sensors 1160 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating components.

[0179] Conventional ADAS systems can be prone to false positives, which can be annoying and distracting for the driver, but typically not catastrophic, as they alert the driver and allow them to determine whether a safe condition truly exists and take appropriate action. However, in the autonomous vehicle 1100, in the event of conflicting results, the vehicle 1100 itself must decide whether to heed the results from the main computer or auxiliary computer (e.g., the first controller 1136 or the second controller 1136). For example, in some embodiments, the ADAS system 1138 may be a backup and / or auxiliary computer for providing perception information to a backup computer rationality module. The backup computer rationality monitor may run redundant and varied software on hardware components to detect faults in perception and dynamic driving tasks. Outputs from the ADAS system 1138 may be provided to a supervisory MCU. If the outputs from the main computer and the auxiliary computer conflict, the supervisory MCU must determine how to reconcile the conflict to ensure safe operation.

[0180] In some examples, the master computer can be configured to provide a confidence score to the supervisory MCU, indicating the master computer's confidence level in the selected result. If the confidence score exceeds a threshold, the supervisory MCU can follow the master computer's direction regardless of whether the auxiliary computer provides conflicting or inconsistent results. If the confidence score does not meet the threshold and the master and auxiliary computers indicate different results (e.g., conflict), the supervisory MCU can arbitrate between these computers to determine the appropriate result.

[0181] The supervisory MCU can be configured to run a neural network trained and configured to determine the conditions under which the auxiliary computer provides a false alarm, based at least in part on outputs from both the host and auxiliary computers. Thus, the neural network in the supervisory MCU can learn when the output of the auxiliary computer can be trusted and when it cannot. For example, when the auxiliary computer is a RADAR-based FCW system, the neural network in the supervisory MCU can learn when the FCW system is identifying a metallic object that is not actually dangerous, such as a drain grid or manhole cover that triggers an alarm. Similarly, when the auxiliary computer is a camera-based LDW system, the neural network in the supervisory MCU can learn to ignore the LDW when a cyclist or pedestrian is present and lane departure is actually the safest strategy. In embodiments that include a neural network running on the supervisory MCU, the supervisory MCU may include at least one of a DLA or GPU suitable for running the neural network using associated memory. In a preferred embodiment, the supervisory MCU may include a component of SoC 1104 and / or be included as a component of SoC 1104.

[0182] In other examples, ADAS system 1138 may include an auxiliary computer that performs ADAS functions using conventional computer vision rules. This allows the auxiliary computer to use classic computer vision rules (if-then), and the presence of neural networks in the supervising MCU can improve reliability, safety, and performance. For example, diverse implementations and intentional non-identity make the entire system more fault-tolerant, especially for failures caused by software (or software-hardware interface) functionality. For instance, if a software vulnerability or bug exists in the software running on the host computer and non-identical software code running on the auxiliary computer provides the same overall result, the supervising MCU can be more confident that the overall result is correct and that the vulnerability in the software or hardware on the host computer does not cause a substantial error.

[0183] In some examples, the output of ADAS system 1138 can be fed to the perception block and / or the dynamic driving task block of the main computer. For example, if ADAS system 1138 issues a forward collision warning because an object is immediately in front, the perception block can use this information when identifying the object. In other examples, the assistance computer can have its own neural network, which is trained and thus reduces the risk of false positives as described herein.

[0184] Vehicle 1100 may also include an infotainment SoC 1130 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system may not be an SoC and may include two or more discrete components. The infotainment SoC 1130 may include a combination of hardware and software that can be used to provide vehicle 1100 with audio (e.g., music, personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., TV, movies, streaming media, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., LTE, WiFi, etc.) and / or information services (e.g., navigation system, rear parking assistance, radio data system, vehicle-related information such as fuel level, total coverage distance, brake fuel level, fuel level, door opening / closing, air filter information, etc.). For example, the infotainment SoC 1130 may include a radio, disc player, navigation system, video player, USB and Bluetooth connectivity, in-vehicle computer, in-vehicle entertainment, WiFi, steering wheel audio controls, hands-free voice controls, head-up display (HUD), HMI display 1134, telematics device, control panel (e.g., for controlling and / or interacting with various components, features, and / or systems) and / or other components. The infotainment SoC 1130 may further be used to provide information (e.g., visual and / or auditory) to users of the vehicle, such as information from the ADAS system 1138, autonomous driving information such as planned vehicle maneuvers, trajectories, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0185] The infotainment SoC 1130 may include GPU functionality. The infotainment SoC 1130 can communicate with other devices, systems, and / or components of the vehicle 1100 via bus 1102 (e.g., CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 1130 may be coupled to a supervisory MCU, allowing the GPU of the infotainment system to perform some autonomous driving functions in the event of a failure of the main controller 1136 (e.g., the primary and / or backup computer of the vehicle 1100). In such an example, the infotainment SoC 1130 may place the vehicle 1100 into a driver-safe parking mode as described herein.

[0186] Vehicle 1100 may also include instrument cluster 1132 (e.g., digital instrument panel, electronic instrument cluster, digital instrument panel, etc.). Instrument cluster 1132 may include a controller and / or supercomputer (e.g., a discrete controller or supercomputer). Instrument cluster 1132 may include a set of instruments such as speedometer, fuel level, oil pressure, tachometer, odometer, turn indicator, shift position indicator, seatbelt warning light, parking brake warning light, engine malfunction indicator, airbag (SRS) system information, lighting controls, safety system controls, navigation information, etc. In some examples, information may be displayed and / or shared between infotainment SoC 1130 and instrument cluster 1132. In other words, instrument cluster 1132 may be included as part of infotainment SoC 1130, or vice versa.

[0187] Figure 11D is a system schematic diagram of communication between a cloud-based server and the example autonomous vehicle 1100 of Figure 11A according to some embodiments of the present disclosure. System 1176 may include server 1178, network 1190, and vehicle 1100. Server 1178 may include multiple GPUs 1184(A)-1284(H) (collectively referred to herein as GPU 1184), PCIe switches 1182(A)-1182(H) (collectively referred to herein as PCIe switch 1182), and / or CPUs 1180(A)-1180(B) (collectively referred to herein as CPU 1180). GPU 1184, CPU 1180, and PCIe switches may be interconnected with high-speed interconnects and / or PCIe connections 1186, such as, but not limited to, NVLink interface 1188 developed by NVIDIA. In some examples, the GPU 1184 is connected via NVLink and / or NVSwitch SoC, and the GPU 1184 and PCIe switch 1182 are connected via PCIe interconnect. Although eight GPUs 1184, two CPUs 1180, and two PCIe switches are illustrated, this is not intended to be limiting. Depending on the embodiment, each of the servers 1178 may include any number of GPUs 1184, CPUs 1180, and / or PCIe switches. For example, each of the servers 1178 may include eight, sixteen, thirty-two, and / or more GPUs 1184.

[0188] Server 1178 can receive image data from vehicles via network 1190, representing images of unexpected or altered road conditions such as recently commenced roadworks. Server 1178 can also transmit neural network 1192, updated neural network 1192, and / or map information 1194, including information about traffic and road conditions, to vehicles via network 1190. Updates to map information 1194 may include updates to HD map 1122, such as information about construction sites, potholes, bends, floods, or other obstacles. In some examples, neural network 1192, updated neural network 1192, and / or map information 1194 may have been generated from new training and / or data received from any number of vehicles in the environment, and / or based on experience gained from training performed at a data center (e.g., using server 1178 and / or other servers).

[0189] Server 1178 can be used to train machine learning models (e.g., neural networks) based on training data. Training data can be generated by the vehicle and / or generated in a simulation (e.g., using a game engine). In some examples, the training data is labeled (e.g., where the neural network benefits from supervised learning) and / or undergoes other preprocessing, while in other examples, the training data is not labeled and / or preprocessed (e.g., where the neural network does not require supervised learning). Training can be performed according to any one or more classes of machine learning techniques, including but not limited to: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, joint learning, transfer learning, feature learning (including principal component and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including alternative dictionary learning), rule-based machine learning, anomaly detection, and any variations or combinations thereof. Once the machine learning model is trained, it can be used by the vehicle (e.g., transmitted to the vehicle via network 1190), and / or the machine learning model can be used by server 1178 to remotely monitor the vehicle.

[0190] In some examples, server 1178 can receive data from a vehicle and apply that data to a state-of-the-art real-time neural network for real-time intelligent inference. Server 1178 may include a deep learning supercomputer powered by GPU 1184 and / or a dedicated AI computer, such as the DGX and DGX Station machines developed by NVIDIA. However, in some examples, server 1178 may include a deep learning infrastructure in a data center that uses only CPU power.

[0191] The deep learning infrastructure of server 1178 may be capable of rapid real-time inference and can be used to assess and verify the health status of the processor, software, and / or associated hardware in vehicle 1100. For example, the deep learning infrastructure may receive periodic updates from vehicle 1100, such as image sequences and / or objects located in those image sequences that vehicle 1100 has already located (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure may run its own neural network to identify objects and compare them with objects identified by vehicle 1100. If the results do not match and the infrastructure concludes that the AI ​​in vehicle 1100 has malfunctioned, then server 1178 may transmit a signal to vehicle 1100 instructing the vehicle's fail-safe computer to take control, notify passengers, and complete a safe stopping operation.

[0192] For inference, server 1178 may include GPU 1184 and one or more programmable inference accelerators (such as NVIDIA's TensorRT). The combination of a GPU-powered server and inference acceleration enables real-time response. In other examples, such as where performance is less critical, CPU, FPGA, and other processor-powered servers can be used for inference.

[0193] Example computing device

[0194] Figure 12 is a block diagram of an example computing device 1200 suitable for implementing some embodiments of the present disclosure. The computing device 1200 may include an interconnect system 1202 directly or indirectly coupled to the following devices: a memory 1204, one or more central processing units (CPUs) 1206, one or more graphics processing units (GPUs) 1208, a communication interface 1210, input / output (I / O) ports 1212, input / output components 1214, a power supply 1216, one or more presentation components 1218 (e.g., one or more displays), and one or more logic units 1220. In at least one embodiment, the computing device 1200 may include one or more virtual machines (VMs), and / or any component thereof may include virtual components (e.g., virtual hardware components). For a non-limiting example, one or more GPUs 1208 may include one or more vGPUs, one or more CPUs 1206 may include one or more vCPUs, and / or one or more logic units 1220 may include one or more virtual logic units. Thus, (one or more) computing devices 1200 may include discrete components (e.g., a full GPU dedicated to computing device 1200), virtual components (e.g., a portion of the GPU dedicated to computing device 1200), or a combination thereof.

[0195] Although the various blocks in Figure 12 are shown connected via interconnect system 1202 using lines, this is not intended to be limiting and is merely for clarity. For example, in some embodiments, presentation component 1218 (such as a display device) may be considered I / O component 1214 (e.g., if the display is a touchscreen). As another example, CPU 1206 and / or GPU 1208 may include memory (e.g., memory 1204 may represent a storage device other than the memory of GPU 1208, CPU 1206, and / or other components). In other words, the computing device of Figure 12 is merely illustrative. No distinction is made between such categories as “workstation,” “server,” “laptop computer,” “desktop computer,” “tablet computer,” “client device,” “mobile device,” “handheld device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types, as all are considered within the scope of the computing device of Figure 12.

[0196] Interconnect system 1202 may represent one or more links or buses, such as address buses, data buses, control buses, or combinations thereof. Interconnect system 1202 may include one or more bus or link types, such as Industry Standard Architecture (ISA) bus, Extended Industry Standard Architecture (EISA) bus, Video Electronics Standards Association (VESA) bus, Peripheral Component Interconnect (PCI) bus, Fast Peripheral Component Interconnect (PCIe) bus, and / or another type of bus or link. In some embodiments, there is a direct connection between components. As an example, CPU 1206 may be directly connected to memory 1204. Further, CPU 1206 may be directly connected to GPU 1208. In cases where there is a direct or point-to-point connection between components, interconnect system 1202 may include a PCIe link to perform the connection. In these examples, a PCI bus is not required to be included in computing device 1200.

[0197] The memory 1204 may include any computer-readable medium from a variety of computer-readable media. The computer-readable medium may be any available medium accessible by the computing device 1200. The computer-readable medium may include volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, the computer-readable medium may include computer storage media and communication media.

[0198] Computer storage media may include volatile and non-volatile media and / or removable and non-removable media implemented with any method or technology for storing information such as computer-readable instructions, data structures, program modules and / or other data types. For example, memory 1204 may store computer-readable instructions (e.g., representing (one or more) programs and / or (one or more) program elements, such as an operating system). Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store the desired information and is accessible by computing device 1200. As used herein, computer storage media does not include the signal itself.

[0199] Computer storage media can embody computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals such as carrier waves or other transmission mechanisms, and includes any information transmission medium. The term "modulated data signal" can refer to a signal whose one or more characteristics are set or altered in a manner that encodes information in the signal. By way of example and not limitation, computer storage media can include wired media (such as wired networks or direct wired connections) and wireless media (such as acoustic, RF, infrared, and other wireless media). Any combination of the above should also be included within the scope of computer-readable media.

[0200] CPU 1206 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 1200 to perform one or more of the methods and / or processes described herein. Each CPU 1206 may contain one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of handling numerous software threads simultaneously. CPU 1206 may contain any type of processor and may contain different types of processors depending on the type of computing device 1200 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 1200, the processor may be an advanced RISC machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplementary coprocessors (such as math coprocessors), computing device 1200 may also include one or more CPUs 1206.

[0201] In addition to or in lieu of one or more CPUs 1206, one or more GPUs 1208 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 1200 to perform one or more of the methods and / or processes described herein. One or more GPUs 1208 may be integrated GPUs (e.g., with one or more CPUs 1206) and / or one or more GPUs 1208 may be discrete GPUs. In embodiments, one or more GPUs 1208 may be coprocessors of one or more CPUs 1206. GPUs 1208 may be used by computing device 1200 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, GPUs 1208 may be used for general-purpose computing on a GPU (GPGPU). GPUs 1208 may contain hundreds or thousands of cores capable of handling hundreds or thousands of software threads simultaneously. GPU 1208 can generate pixel data for an output image in response to rendering commands (e.g., rendering commands received from CPU 1206 via a host interface). GPU 1208 may include graphics memory (e.g., display memory) for storing pixel data or any other suitable data (e.g., GPGPU data). Display memory may be included as part of memory 1204. GPU 1208 may include two or more GPUs operating in parallel (e.g., via links). The links may directly connect the GPUs (e.g., using NVLINK) or may connect the GPUs via a switch (e.g., using NVSwitch). When combined, each GPU 1208 may generate pixel data or GPGPU data for different portions of the output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory or may share memory with other GPUs.

[0202] In addition to or in lieu of CPU 1206 and / or GPU 1208, logic unit 1220 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 1200 to perform one or more of the methods and / or processes described herein. In embodiments, one or more CPUs 1206, one or more GPUs 1208, and / or one or more logic units 1220 may perform any combination of methods, processes, and / or portions thereof discretely or jointly. One or more logic units 1220 may be a portion of one or more CPUs 1206 and / or GPUs 1208 and / or integrated into one or more CPUs 1206 and / or GPUs 1208, and / or one or more logic units 1220 may be discrete components or otherwise external to CPUs 1206 and / or GPUs 1208. In an embodiment, one or more of the logic units 1220 may be coprocessors of one or more of the CPU 1206 and / or one or more of the GPU 1208.

[0203] Examples of logic unit 1220 include one or more processing cores and / or components thereof, such as data processing unit (DPU), tensor core (TC), tensor processing unit (TPU), pixel vision core (PVC), vision processing unit (VPU), graphics processing cluster (GPC), texture processing cluster (TPC), streaming multiprocessor (SM), tree lateral unit (TTU), artificial intelligence accelerator (AIA), deep learning accelerator (DLA), arithmetic logic unit (ALU), application-specific integrated circuit (ASIC), floating-point unit (FPU), input / output (I / O) element, peripheral component interconnect (PCI) or fast peripheral component interconnect (PCIe) element, etc.

[0204] Communication interface 1210 may include one or more receivers, transmitters, and / or transceivers enabling computing device 1200 to communicate with other computing devices via electronic communication networks (including wired and / or wireless communications). Communication interface 1210 may include components and functions for enabling communication over any of a plurality of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., via Ethernet or wirelessband communication), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, one or more logic units 1220 and / or communication interface 1210 may include one or more data processing units (DPUs) for directly transmitting data received via a network and / or via interconnect system 1202 to one or more GPUs 1208 (e.g., their memory).

[0205] I / O port 1212 enables computing device 1200 to be logically coupled to other devices including I / O component 1214, (one or more) presentation component 1218, and / or other components, some of which may be built into (e.g., integrated into) computing device 1200. Illustrative I / O component 1214 includes microphones, mice, keyboards, joysticks, gamepads, game controllers, satellite dish antennas, scanners, printers, wireless devices, etc. I / O component 1214 can provide a natural user interface (NUI) that processes aerial gestures, voice, or other physiological input generated by the user. In some cases, input may be transmitted to appropriate network elements for further processing. The NUI can implement any combination of voice recognition, pen recognition, facial recognition, biometric recognition, on-screen and near-screen gesture recognition, aerial gestures, head and eye tracking, and touch recognition (as described in more detail below) associated with the display of computing device 1200. Computing device 1200 may include depth cameras for gesture detection and recognition, such as stereo camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof. Additionally, the computing device 1200 may include an accelerometer or gyroscope (e.g., as part of an inertial measurement unit (IMU)) that enables motion detection. In some examples, the computing device 1200 may use the output of the accelerometer or gyroscope to render immersive augmented reality or virtual reality.

[0206] Power supply 1216 may include a hardwired power supply, a battery power supply, or a combination thereof. Power supply 1216 may provide power to computing device 1200 to enable the components of computing device 1200 to operate.

[0207] The presentation component 1218 may include a display (e.g., a monitor, touchscreen, television screen, head-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. The presentation component 1218 may receive data from other components (e.g., GPU 1208, CPU 1206, DPU, etc.) and output the data (e.g., as images, videos, sounds, etc.).

[0208] Example Data Center

[0209] Figure 13 illustrates an example data center 1300 that may be used in at least one embodiment of this disclosure. Data center 1300 may include a data center infrastructure layer 1310, a framework layer 1320, a software layer 1330, and / or an application layer 1340.

[0210] As shown in Figure 13, the data center infrastructure layer 1310 may include a resource coordinator 1312, grouped computing resources 1314, and node computing resources (“nodes CRs”) 1316(1)-1316(N), where “N” represents any complete positive integer. In at least one embodiment, nodes CRs 1316(1)-1316(N) may include, but are not limited to, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field-programmable gate arrays (FPGAs), graphics processing units or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output (NW I / O) devices, network switches, virtual machines (VMs), power modules and / or cooling modules, etc. In some embodiments, one or more node CRs from nodes CRs 1316(1)-1316(N) may correspond to servers having one or more of the aforementioned computing resources. In addition, in some embodiments, nodes CRs1316(1)-13161(N) may include one or more virtual components, such as vGPU, vCPU, etc., and / or one or more nodes CRs1316(1)-1316(N) may correspond to virtual machines (VMs).

[0211] In at least one embodiment, the grouped computing resources 1314 may include individual groups of node CRs 1316 housed within one or more racks (not shown), or a plurality of racks housed within a data center in different geographical locations (also not shown). Individual groups of node CRs 1316 within the grouped computing resources 1314 may include grouped computing, networking, memory, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, a plurality of node CRs 1316, including CPUs, GPUs, DPUs, and / or other processors, may be grouped within one or more racks to provide computing resources to support one or more workloads. One or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0212] Resource coordinator 1312 may be configured or otherwise control one or more nodes CRs 1316(1)-1316(N) and / or grouped computing resources 1314. In at least one embodiment, resource coordinator 1312 may include a Software Design Infrastructure (SDI) management entity for data center 1300. Resource coordinator 1312 may include hardware, software, or some combination thereof.

[0213] In at least one embodiment, as shown in FIG13, the framework layer 1320 may include a job scheduler 1333, a configuration manager 1334, a resource manager 1336, and / or a distributed file system 1338. The framework layer 1320 may include a framework for software 1332 supporting the software layer 1330 and / or one or more applications 1342 supporting the application layer 1340. Software 1332 or application 1342 may respectively comprise web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. The framework layer 1320 may be, but is not limited to, a free and open-source software web application framework (such as Apache Spark) capable of large-scale data processing (e.g., "big data") using the distributed file system 1338. TM(Hereinafter referred to as "Spark") is a type of resource manager. In at least one embodiment, the job scheduler 1333 may include Spark drivers to facilitate the scheduling of workloads supported by different layers of data center 1300. The configuration manager 1334 may be able to configure different layers, such as software layer 1330 and framework layer 1320 (which includes Spark and distributed file system 1338 for supporting large-scale data processing). The resource manager 1336 may be able to manage compute resources mapped to or allocated to clusters of distributed file system 1338 and job scheduler 1333 or to support clusters of distributed file system 1338 and job scheduler 1333. In at least one embodiment, the clustered or grouped compute resources may include grouped compute resources 1314 in data center infrastructure layer 1310. The resource manager 1336 may coordinate with resource coordinator 1312 to manage these mapped or allocated compute resources.

[0214] In at least one embodiment, the software 1332 included in software layer 1330 may include software used in at least a portion of the nodes CRs 1316(1)-1316(N), the grouped computing resources 1314, and / or the distributed file system 1338 of framework layer 1320. One or more types of software may include, but are not limited to, internet web search software, email virus scanning software, database software, and streaming video content software.

[0215] In at least one embodiment, the application 1342 included in the application layer 1340 may include one or more types of applications used at least in part by nodes CRs 1316(1)-1316(N), grouped computing resources 1314, and / or the distributed file system 1338 of the framework layer 1320. One or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in combination with one or more embodiments.

[0216] In at least one embodiment, any of the configuration manager 1334, resource manager 1336, and resource coordinator 1312 can implement any number and type of self-modification actions based on any amount and type of data acquired in any technically feasible manner. Self-modification actions can free data center operators of data center 1300 from making potentially poor configuration decisions and may prevent underutilization and / or poor performance of the data center.

[0217] According to one or more embodiments described herein, data center 1300 may include tools, services, software, or other resources to train one or more machine learning models or to use one or more machine learning models to predict or infer information. For example, one or more machine learning models can be trained by using the software and / or computing resources described above with respect to data center 1300 to compute weight parameters according to a neural network architecture. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks can be used to infer or predict information using the resources described above with respect to data center 1300 by using weight parameters computed through one or more training techniques (such as, but not limited to, those described herein).

[0218] In at least one embodiment, the data center 1300 may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, and / or other hardware (or corresponding virtual computing resources) to perform training and / or inference using the aforementioned resources. Furthermore, one or more of the software and / or hardware resources described above may be configured to allow a user to train or perform services that infer information, such as image recognition, speech recognition, or other artificial intelligence services.

[0219] Example network environment

[0220] A network environment suitable for implementing embodiments of this disclosure may include one or more client devices, servers, network-attached storage (NAS), other back-end devices, and / or other device types. Client devices, servers, and / or other device types (e.g., each device) may be implemented on one or more instances of computing devices 1200 of FIG. 12—for example, each device may include similar components, features, and / or functions of computing devices 1200. Furthermore, in the case of implementing back-end devices (e.g., servers, NAS, etc.), the back-end devices may be included as part of a data center 1300, examples of which are described in more detail herein with reference to FIG. 13.

[0221] Components of a network environment can communicate with each other via a network, which can be wired, wireless, or both. A network can include multiple networks or one of multiple networks. For example, a network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. Where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) can provide wireless connectivity.

[0222] A compatible network environment may include one or more peer-to-peer network environments (in which case the server may not be included in the network environment) and one or more client-server network environments (in which case one or more servers may be included in the network environment). In a peer-to-peer network environment, the functionality described herein for the server can be implemented on any number of client devices.

[0223] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, etc. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework supporting software at the software layer and / or application at the application layer. The software or application may respectively include network-based service software or applications. In embodiments, one or more client devices may use the network-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a free and open-source software network application framework that can use a distributed file system for large-scale data processing (e.g., "big data").

[0224] A cloud-based network environment can provide cloud computing and / or cloud storage for any combination of the computing and / or data storage functions (or one or more portions thereof) described herein. Any of these different functions can be distributed across multiple locations from a central or core server (e.g., distributed across one or more data centers at the state, region, country, global, etc.). The core server may assign at least a portion of the functionality to the edge server if the connection to the user (e.g., a client device) is relatively close to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0225] One or more client devices may include at least some of the components, features, and functions of one or more example computing devices 1200 described herein with respect to FIG. 12. By way of example and not limitation, client devices may be implemented as personal computers (PCs), laptops, mobile devices, smartphones, tablets, smartwatches, wearable computers, personal digital assistants (PDAs), MP3 players, virtual reality headsets, global positioning systems (GPS) or devices, video players, cameras, surveillance equipment or systems, vehicles, ships, spacecraft, virtual machines, drones, robots, handheld communication devices, hospital equipment, gaming devices or systems, entertainment systems, vehicle computer systems, embedded system controllers, remote controls, electrical appliances, consumer electronics devices, workstations, edge devices, any combination of these depicted devices, or any other suitable device.

[0226] This disclosure can be described in the general context of machine-usable instructions or computer code, including computer-executable instructions such as program modules, which are executed by a computer or other machine such as a personal digital assistant or other handheld device. Typically, a program module, including routines, programs, objects, components, data structures, etc., refers to code that performs a specific task or implements a specific abstract data type. This disclosure can be practiced in a wide variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc. This disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices linked via a communication network.

[0227] As used herein, the phrase "and / or" relating to two or more elements should be interpreted as referring to only one element or a combination of elements. For example, "element A, element B, and / or element C" can include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or element A, B, and C. Furthermore, "at least one of element A or element B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Further, "at least one of element A and element B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0228] The subject matter of this disclosure is described in detail herein to satisfy statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the discloser has envisioned that the claimed subject matter may be embodied in other ways to include steps different from or similar combinations of steps described herein in conjunction with other current or future techniques. Moreover, although the terms “step” and / or “box” may be used herein to imply different elements of the method employed, these terms should not be construed as implying any particular order among or between the various steps disclosed herein, unless the order of the steps is explicitly described.

[0229] Example paragraph

[0230] A. A method comprising: applying input data to one or more machine learning models, the input data comprising at least: first feature data representing at least one of first geometric information or first semantic information corresponding to a plurality of lane segments; and second feature data representing at least one of second geometric information or second semantic information corresponding to a plurality of traffic control signals; calculating a plurality of confidence scores using the one or more machine learning models and at least based on the input data, the confidence scores indicating whether one or more activation signals of the plurality of traffic control signals correspond to one or more lane segments of the plurality of lane segments; associating the one or more activation signals with the one or more lane segments at least based on the plurality of confidence scores; and causing a machine to perform one or more operations at least based on the association.

[0231] B. The method as described in paragraph A, wherein the first semantic information corresponding to the plurality of lane segments includes at least one or more directions associated with the plurality of lane segments.

[0232] C. The method as described in paragraph A, wherein the one or more machine learning models comprise one or more deep neural networks.

[0233] D. The method as described in paragraph C further comprises: synthetically generating one or more training datasets from non-visual data using one or more rules associated with the positioning of traffic control signal equipment relative to one or more corresponding lane segments; and training the one or more deep neural networks using at least in part the one or more training datasets.

[0234] E. The method as described in paragraph A, wherein associating the one or more activation signals with the one or more lane segments comprises at least: associating one or more first activation signals with one or more first lane segments having one or more first directions; and associating one or more second activation signals with one or more second lane segments having one or more second directions.

[0235] F. The method as described in paragraph A, wherein the first feature data and the second feature data include non-image features.

[0236] G. The method as described in paragraph A further comprises: comparing a first confidence score associated with a first pairing between a first activation signal and a first lane segment with one or more second confidence scores associated with one or more second pairs between one or more second activation signals and one or more second lane segments; determining, at least based on the comparison, that the first confidence score is greater than the one or more second confidence scores by more than a threshold; and associating the first activation signal and the first lane segment as a valid pairing, at least based on the first confidence score being greater than the one or more second confidence scores by more than the threshold.

[0237] H. A system comprising: one or more processors configured to: apply at least the following to one or more machine learning models: one or more first features corresponding to a plurality of lanes; and one or more second features corresponding to a plurality of traffic control devices; process the one or more first features and the one or more second features based at least on the one or more machine learning models; associate at least one of the plurality of traffic control devices with at least one of the plurality of lanes; and cause the machine to perform one or more control operations based at least on the association.

[0238] I. The system as described in paragraph H, wherein the one or more processors are further configured to: synthesize at least one of training data or ground truth data from non-visual data using one or more rules associated with locating one or more traffic control devices relative to one or more corresponding lane segments.

[0239] J. The system as described in paragraph H, wherein the one or more machine learning models comprise one or more deep neural networks (DNNs).

[0240] K. The system as described in paragraph H, wherein the one or more first features and the one or more second features are non-image features.

[0241] L. The system as described in paragraph H, wherein the one or more first features indicate at least one of the following: one or more geometries associated with the plurality of lanes; or one or more directions associated with the plurality of lanes.

[0242] M. The system as described in paragraph H, wherein the one or more second features indicate at least one of the following: one or more geometries associated with the plurality of traffic control devices; one or more road user classifications associated with the plurality of traffic control devices; or one or more states associated with the plurality of traffic control devices.

[0243] N. The system as described in paragraph H, wherein the one or more processors are further configured to: calculate a plurality of scores using the one or more machine learning models and based at least on the one or more first features and the one or more second features, the plurality of scores indicating whether the traffic control device corresponds to the lane or one or more second lanes among the plurality of lanes; and determine that the traffic control device corresponds to the lane based at least on the first score being greater than one or more second scores among the plurality of scores, wherein the association between the traffic control device and the lane is based at least on the first score being greater than the one or more second scores.

[0244] O. The system as described in paragraph H, wherein the association of the traffic control device with the lane includes processing at least one or more first features and one or more second features based on the one or more machine learning models, and associating one or more luminous signals of the traffic control device with one or more directions associated with the lane.

[0245] P. As described in paragraph H, the one or more processors are further configured to: generate a binary matrix using the one or more machine learning models, the binary matrix including a plurality of entries indicating one or more valid pairings between a respective traffic control device among the plurality of traffic control devices and a respective lane among the plurality of lanes; and wherein the association between at least the traffic control device and the lane is based at least on the binary matrix.

[0246] Q. A system as described in paragraph H, wherein the system comprises at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulated operations; a system for performing one or more digital twin operations; a system for performing optical transmission simulation; a system for performing collaborative content creation of 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more generative AI operations; a system for performing operations using a large language model; a system for performing operations using one or more visual language models (VLMs); a system for performing operations using one or more multimodal language models; a system for using or deploying one or more inference microservices; a system for performing one or more conversational AI operations; a system for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system comprising one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

[0247] R. One or more processing units, the one or more processing units comprising: processing circuitry for updating one or more parameters of one or more deep neural networks (DNNs) to train the one or more DNNs using a training dataset to associate one or more activated traffic control signals with one or more directions of one or more lanes on a driving road surface, the training dataset being synthetically generated from non-visual data based at least on one or more rules associated with locating one or more traffic control devices relative to one or more corresponding lane segments.

[0248] S. One or more processing units as described in paragraph R, wherein the training dataset is synthetically generated at least by: projecting the three-dimensional 3D geometry of multiple lanes into a two-dimensional 2D image space based at least on one or more internal camera parameters or external camera parameters; projecting the three-dimensional 3D geometry of multiple traffic control devices into a two-dimensional 2D image space based at least on the one or more internal camera parameters or external camera parameters; and generating multiple traffic control device-lane pairings between the multiple lanes and the multiple traffic control devices based at least on one or more traffic rules.

[0249] T. One or more processing units as described in paragraph R, wherein the one or more processing units are included in at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing optical transmission simulation; a system for performing collaborative content creation of 3D assets; a system for performing one or more deep learning operations; a system implemented using edge devices; a system implemented using robots; a system for performing one or more generative AI operations; a system for performing operations using large language models; a system for performing operations using one or more visual language models (VLMs); a system for performing operations using one or more multimodal language models; a system for using or deploying one or more inference microservices; a system for performing one or more conversational AI operations; a system for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system containing one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

Claims

1. A method, the method comprising: The input data is applied to one or more machine learning models, the input data comprising at least: first feature data representing at least one of first geometric information or first semantic information corresponding to a plurality of lane segments; and second feature data representing at least one of second geometric information or second semantic information corresponding to a plurality of traffic control signals; using the one or more machine learning models and at least based on the input data to compute a plurality of confidence scores, the confidence scores indicating whether one or more activation signals of the plurality of traffic control signals correspond to one or more lane segments of the plurality of lane segments; associating the one or more activation signals with the one or more lane segments at least based on the plurality of confidence scores; and causing the machine to perform one or more operations at least based on the association.

2. The method as described in claim 1, wherein, The first semantic information corresponding to the plurality of lane segments includes at least one or more directions associated with the plurality of lane segments.

3. The method as described in claim 1, wherein, The one or more machine learning models include one or more deep neural networks.

4. The method of claim 3, further comprising: One or more training datasets are synthetically generated from non-visual data using one or more rules associated with the positioning of traffic control signal equipment relative to one or more corresponding lane segments. And at least in part using the one or more training datasets to train the one or more deep neural networks.

5. The method of claim 1, wherein, Associating the one or more activation signals with the one or more lane segments includes at least: associating one or more first activation signals with one or more first lane segments having one or more first directions; and associating one or more second activation signals with one or more second lane segments having one or more second directions.

6. The method of claim 1, wherein, The first feature data and the second feature data include non-image features.

7. The method of claim 1, further comprising: A first confidence score associated with a first pairing between a first activation signal and a first lane segment is compared with one or more second confidence scores associated with one or more second pairs between one or more second activation signals and one or more second lane segments; based at least on the comparison, it is determined that the first confidence score is greater than the one or more second confidence scores by more than a threshold; and the first activation signal and the first lane segment are associated as a valid pairing based at least on the fact that the first confidence score is greater than the one or more second confidence scores by more than the threshold.

8. A system comprising: One or more processors are used to: apply at least the following to one or more machine learning models: one or more first features corresponding to multiple lanes; The system includes: a first feature corresponding to one or more traffic control devices; processing the first feature and the second feature based on at least one or more machine learning models; associating at least one traffic control device with at least one lane of the multiple traffic control devices; and causing the machine to perform one or more control operations based on at least the association.

9. The system of claim 8, wherein the one or more processors are further configured to: synthesize at least one of training data or ground truth data from non-visual data using one or more rules associated with locating one or more traffic control devices relative to one or more corresponding lane segments.

10. The system of claim 8, wherein, The one or more machine learning models include one or more deep neural networks (DNNs).

11. The system of claim 8, wherein, The one or more first features and the one or more second features are non-image features.

12. The system of claim 8, wherein, The one or more first features indicate at least one of the following: one or more geometries associated with the plurality of lanes; or one or more directions associated with the plurality of lanes.

13. The system of claim 8, wherein, The one or more second features indicate at least one of the following: one or more geometries associated with the plurality of traffic control devices; one or more road user classifications associated with the plurality of traffic control devices; or one or more states associated with the plurality of traffic control devices.

14. The system of claim 8, wherein the one or more processors are further configured to: calculate a plurality of scores using the one or more machine learning models and based at least on the one or more first features and the one or more second features, the plurality of scores indicating whether the traffic control device corresponds to the lane or one or more second lanes among the plurality of lanes; and determine that the traffic control device corresponds to the lane based at least on the fact that a first score among the plurality of scores is greater than one or more second scores among the plurality of scores, wherein, The association between the traffic control device and the lane is based at least on the first score being greater than one or more of the second scores.

15. The system of claim 8, wherein, The association of the traffic control device with the lane includes processing at least one or more first features and one or more second features based on one or more machine learning models, and associating one or more luminous signals of the traffic control device with one or more directions associated with the lane.

16. The system of claim 8, wherein the one or more processors are further configured to: generate a binary matrix using the one or more machine learning models, the binary matrix comprising a plurality of entries indicating one or more valid pairings between a respective traffic control device among the plurality of traffic control devices and a respective lane among the plurality of lanes; and wherein, The association between the traffic control device and the lane is at least based on the binary matrix.

17. The system of claim 8, wherein the system comprises at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulated operations; a system for performing one or more digital twin operations; a system for performing optical transmission simulation; a system for performing collaborative content creation of 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more generative AI operations; a system for performing operations using a large language model; a system for performing operations using one or more visual language models (VLMs); a system for performing operations using one or more multimodal language models; a system for using or deploying one or more inference microservices; a system for performing one or more conversational AI operations; a system for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system comprising one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

18. One or more processing units, said one or more processing units comprising: A processing circuit for updating one or more parameters of one or more deep neural networks (DNNs) to train the one or more DNNs to associate one or more active traffic control signals with one or more directions of one or more lanes on a driving road surface using a training dataset, the training dataset being synthetically generated from non-visual data based at least on one or more rules associated with locating one or more traffic control devices relative to one or more corresponding lane segments.

19. The processing unit of claim 18, wherein the training dataset is synthetically generated at least by: projecting the three-dimensional 3D geometry of a plurality of lanes into a two-dimensional 2D image space based at least on one or more internal camera parameters or external camera parameters; projecting the three-dimensional 3D geometry of a plurality of traffic control devices into a two-dimensional 2D image space based at least on the one or more internal camera parameters or external camera parameters; and generating a plurality of traffic control device-lane pairings between the plurality of lanes and the plurality of traffic control devices based at least on one or more traffic rules.

20. The one or more processing units as claimed in claim 18, wherein, The one or more processing units are included in at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing optical transmission simulation; a system for performing collaborative content creation of 3D assets; a system for performing one or more deep learning operations; a system implemented using edge devices; a system implemented using robots; a system for performing one or more generative AI operations; a system for performing operations using large language models; a system for performing operations using one or more visual language models (VLMs); a system for performing operations using one or more multimodal language models; a system for using or deploying one or more inference microservices; a system for performing one or more conversational AI operations; a system for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system containing one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

Images