TRAFFIC LIGHT CLASSIFICATION FOR AUTONOMOUS AND SEMI-AUTOMATIC SYSTEMS AND APPLICATIONS

Abstract

In various examples, the systems and methods of this disclosure can train and use machine learning models to determine attributes and, in some cases, classifications related to traffic lights in order to establish traffic rules for the operation of a machine (e.g., an autonomous or semi-autonomous machine or vehicle) in an environment. For example, an image representing a traffic light device can be applied to a machine learning model that includes a variety of component heads. Each of the component heads can be trained to recognize different attributes and / or combinations of attributes associated with the traffic light device. Additionally, in some examples, the machine learning model can include a fusion head trained to classify the traffic light device.For example, the fusion head can classify the traffic light device using the detected attributes and / or using a combined feature vector from multiple feature vectors applied to the multitude of component heads.

Description

BACKGROUND

[0001] For an autonomous or semi-autonomous vehicle to navigate safely through an environment, it may need to be able to correctly determine the status of traffic lights. This capability helps the vehicle understand the traffic rules in effect at a particular location or intersection. However, the physical characteristics (e.g., appearance) of traffic lights typically vary from one geographic region to another. Traffic lights in different regions may have different shapes, orientations, numbers of bulbs, bulb colors, bulb shapes, and / or other physical features. Therefore, in many scenarios, it can be difficult to correctly identify the traffic rules conveyed by the state of a particular traffic light. SUMMARY

[0002] The invention is defined by the claims. To illustrate the invention, aspects and embodiments are described here that may or may not fall within the scope of the claims.

[0003] Embodiments of the present disclosure relate to the classification of traffic lights for autonomous or semi-autonomous systems and applications. Systems and methods are disclosed that can train and use machine learning models to determine attributes and, in some cases, classifications associated with traffic lights in order to establish traffic rules for the operation of a machine (e.g., an autonomous or semi-autonomous machine or vehicle) in an environment. For example, an image representing a traffic light device can be applied to a machine learning model comprising a plurality of component heads. Each component head of the plurality of component heads can be trained to recognize various attributes and / or combinations of attributes associated with the traffic light device, such as...Active lamp colors and / or shapes, number of lamps, housing orientation, and / or other attributes. In some examples, the machine learning model may include a fusion head trained to classify the traffic light device. The fusion head may, for example, classify the traffic light device using the recognized attributes or embeddings from the plurality of component heads and / or using a combined feature vector of several feature vectors applied to the plurality of component heads. Using the recognized attributes and / or the classification of the traffic light device, the systems of this disclosure may cause the machine to perform one or more control operations.

[0004] In contrast to conventional systems, the systems of the present disclosure are in some embodiments able to use a multi-component machine learning architecture to classify each component (where, for example, each component can represent one or more attributes of a traffic light), and in some cases to use a fusion head or classifier to fuse the features from each component head to predict a final class of the traffic light.For example, the systems of the present disclosure can use a multi-component machine learning model to decompose a traffic light into several components, where the active light state can be one of the components, and then a fusion head can be used within the model to predict the final traffic light class by combining all of these components, which in some cases allows cross-checking between components and the removal or reduction of post-processing.Furthermore, unlike conventional systems, the systems of the present disclosure can apply implicit negative training targets to the machine learning models for each negative example during training and assign the negative examples to a uniform distribution, thereby enabling the models to better distinguish valid examples from unknown or invalid examples, thus reducing false-positive activations in each component and during fusion.

[0005] Further features of the disclosure are characterized by the independent and dependent claims.

[0006] Any feature in one aspect of the disclosure can be applied in any suitable combination to other aspects of the disclosure. In particular, methods can be applied to device or system aspects and vice versa.

[0007] Furthermore, features implemented in hardware can be implemented in software and vice versa. Any reference to software and hardware features in this description should be interpreted accordingly.

[0008] Each system or device feature described herein can also be provided as a procedure, and vice versa. Functionally described system and / or device aspects (including means-plus functional features) can alternatively be expressed in terms of their corresponding structure, for example, as a suitably programmed processor and associated memory.

[0009] It should also be noted that certain combinations of the various features described and defined in any aspect of the revelation can be implemented and / or provided and / or used independently of one another.

[0010] The disclosure also includes computer programs and computer program products containing software code adapted to perform one of the methods when executed on a data processing device and / or embody one of the device and system features described herein, including all or part of the individual steps of a method.

[0011] The disclosure also provides a computer or computer system (including networked or distributed systems) with an operating system that supports a computer program for carrying out one of the procedures described herein and / or for embodying one of the device or system features described herein.

[0012] The revelation also provides a computer-readable medium on which one or more of the aforementioned computer programs are stored.

[0013] The disclosure also includes a signal that transmits one or more of the aforementioned computer programs.

[0014] The disclosure extends to methods and / or devices and / or systems as described herein with reference to the accompanying drawings.

[0015] Aspects and embodiments of the revelation will now be described purely by way of example with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS The present systems and methods for traffic light classification for autonomous or semi-autonomous systems and applications are described in detail below with reference to the attached figures, wherein Fig. 1 is a data flow diagram illustrating an example of a process for classifying traffic lights using a multi-component machine learning model according to some embodiments of the present disclosure; Fig. 2 is a representation of a section of an image showing a traffic light, wherein the section of the image was obtained from image data representing an image of an environment, according to some embodiments of the present disclosure; Fig. Three different examples of traffic light configurations and states are shown, which can be classified using a multi-component machine learning model according to some embodiments of the present disclosure; Fig. Figure 4 illustrates an example of the use of a multi-component machine learning architecture to predict attributes and / or classifications of traffic lights from image data according to some embodiments of the present disclosure; Fig. 5 is a data flow diagram illustrating an example procedure for training one or more machine learning models to predict traffic light classes and / or attributes according to some embodiments of the present disclosure; Fig. Figure 6 illustrates an example of a system that can perform one or more of the processes described herein according to some embodiments of the present disclosure; Fig. Figure 7 is a flowchart illustrating an example of a method for classifying traffic lights according to some embodiments of the present disclosure. Fig. Figure 8 is a flowchart illustrating an example of a method for determining traffic light attributes using a multi-component machine learning model according to some embodiments of the present disclosure. Fig. Figure 9 is a flowchart illustrating an example of a method for training a multi-component machine learning model to predict traffic light attributes and / or classes according to some embodiments of the present disclosure; Fig. Figure 10A is a representation of an example of an autonomous vehicle according to some embodiments of the present disclosure; Fig. 10B is an example of camera positions and fields of view for the autonomous vehicle from Fig. 10A according to some embodiments of the present disclosure; Fig. 10C is a block diagram of an example system architecture for the example of an autonomous vehicle from Fig. 10A according to some embodiments of the present disclosure; Fig. 10D is a system diagram for communication between one or more cloud-based servers and the example of an autonomous vehicle from Fig. 10A according to some embodiments of the present disclosure; Fig. Figure 11 is a block diagram of an example computer device suitable for use in implementing some embodiments of the present disclosure; and Fig. Figure 12 is a block diagram of an example data center suitable for use in the implementation of some embodiments of the present disclosure. DETAILED DESCRIPTION

[0016] Systems and methods related to the classification of traffic lights for autonomous or semi-autonomous systems and applications are disclosed. Although the present disclosure is based on an example of an autonomous or semi-autonomous vehicle or machine 1000 (hereafter referred to alternatively as "Vehicle 1000", "Ego-Vehicle 1000" or "Machine 1000", "Ego-Machine 1000" or "Machine 1000"), an example of which is given in relation to the Fig. The fact that the systems and procedures described in sections 10A-10D are not described herein is not intended to be a limitation. For example, the systems and procedures described herein may be used without restriction by non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robotic platforms, autonomous mobile robots (AMRs), humanoid robots, warehouse vehicles, all-terrain vehicles, vehicles coupled with one or more trailers, aircraft, boats, shuttles, emergency vehicles, motorcycles, electric or motorcycles, airplanes, construction vehicles, underwater vehicles, drones, and / or other types of vehicles.Furthermore, although the present disclosure can be described in relation to traffic light classification for autonomous or semi-autonomous driving, this is not intended to be a limitation, and the systems and methods described herein can be used in the fields of augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications and / or other technology areas where attribute-based classification can be used.

[0017] As described above, traffic lights can look very different in different regions. For example, each light source in a traffic light can have a different shape and color, and a traffic light can have multiple active light sources (e.g., illuminated, switched on, etc.) simultaneously. Although traffic lights can vary greatly in different regions, they generally all consist of a relatively small number of components. For this reason, and as described herein, a concise model architecture can be used to map an input image or image section of a traffic light.Instead of directly predicting a final traffic light class, for example, the models trained and used by the systems of this disclosure can predict classes of different components of a traffic light using a common backbone and different component heads, where each component head can have one or more valid outputs. In some examples, each component can represent or correspond to one or more attributes of the traffic light. Examples of attributes include, but are not limited to, the orientation of the traffic light, the road users to whom the traffic light applies (e.g., vehicles, pedestrians, cyclists, etc.), the shape of the traffic light housing, whether a traffic light is obscured, the color(s) of the active light source(s), the shape(s) of the active light source(s), or other attributes.In some cases, the systems can use a fusion head of the model(s) to combine embeddings from the various component heads and output a final traffic light class. Furthermore, in some examples, the systems of this disclosure can apply implicit negative training targets for each negative example to reduce false-positive activations in each component head and / or in the fusion head. For example, the systems of this disclosure can assign negative examples to a uniform distribution, which can help the model(s) better distinguish valid examples from unknown or invalid examples.

[0018] For example, but not limited to, a system can capture image data representing an image (e.g., a cropped section of a larger image) showing a traffic light, and the system can apply the image to one or more machine learning models (e.g., one or more deep neural networks (DNNs)). Although examples relating to the use of neural networks, and DNNs in particular, in machine learning models are described here, this is not intended to be a limitation.For example, and without limitation, any of the various machine learning models and / or neural networks described herein can include any type of machine learning model, such as a machine learning model that uses linear regression, logistic regression, decision trees, support vector machines (SVMs), Naive Bayes, k-nearest neighbors (Knn), K-means clustering, random forest, dimensionality reduction algorithms, gradient enhancement algorithms, neural networks (e.g.,Autoencoder neural networks, artificial neural networks (ANNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), perceptrons, long / short term memory (LSTM) networks, multilayer perceptron (MLP) networks, deep stacking networks (DSNs), generative pre-training (GPT) models or networks, feedforward networks, radial basis function ANNs, self-organizing maps (SOMs), Kohonen maps, Hopfield networks, Boltzmann machines, deep belief neural networks, deconvolutional neural networks, generative adversarial networks (GANs), liquid-state machines, modular neural networks, liquid-state machines, sequence-to-sequence models, networks with transformer architectures, diffusion models (e.g., probabilistic diffusion models, score-based generative models, etc.), neural rendering field models (NeRF), models with pure encoder architectures, models with pure decoder architectures, models with encoder-decoder architectures, generative machine learning models, language models, large language models (LLMs), etc.). etc.), Neural rendering field (NeRF) models, models with pure encoder architectures, models with pure decoder architectures, models with encoder-decoder architectures, generative machine learning models, language models, large language models (LLMs), image-language models (VLMs), multimodal language models (MMLMs), etc.) and / or other types of machine learning models.

[0019] As described herein, the machine learning models in some examples may incorporate a multi-component architecture. That is, the machine learning models may include a variety of component heads, and each component head may be trained to recognize one or more attributes of the traffic light depicted in the image. For example, the machine learning models may include a first component head trained to predict one or more first attributes of the traffic light (e.g., orientation), a second component head trained to predict one or more second attributes of the traffic light (e.g., housing shape), a third component head trained to predict one or more third attributes of the traffic light (e.g., color(s) and / or shape(s) of the active light source(s)), and so on.In various examples, a single component head can have more than one valid output for a given input image. For instance, a component head designed to predict active light source shapes can output a message indicating that the active light sources of a traffic light include a round light source and an arrow light source.

[0020] In some examples, the machine learning model(s) may include a common backbone that generates a common feature vector from the input image. The machine learning model(s) can extract various features from the common feature vector to generate component-specific feature vectors to be applied to each of the model's component heads. For example, the machine learning models may use the common feature vector to generate a first component feature vector to be applied to the first component head of the model, a second component feature vector to be applied to the second component head of the model, and so on. In such examples, the features contained in the component-specific feature vectors can enable the component heads to make accurate predictions.In other words, component-specific feature vectors can include features from the common feature vector that the component head can use to predict attributes it was trained to identify, while excluding features from the common feature vector that the component head cannot use. For example, a component feature vector applied to a component head to predict the orientation of a traffic light can exclude features related to active light source colors, and vice versa.

[0021] In various examples, the machine learning models can apply the component-specific feature vectors to the component heads, and the component heads can output probability vectors that specify the attributes associated with the traffic light device represented in the input image. For example, a probability vector for active light color attributes can contain entries that indicate the probability or reliability of the active light color for the traffic light. For instance, if the active light color is green, the entries in the probability vector for the light being red or yellow can have a low value (e.g., 1%), while the entry for the light being green can have a higher value (e.g., 98%).In some examples, the traffic light attributes that the component heads can predict may include orientation, housing shape, active light color, active light shape, number of lights, obstruction (e.g., whether the traffic light is obscured by other objects), clipping (e.g., where the traffic light is cut off by the image border), road users (e.g., for which type(s) of road users the traffic light applies), and / or flashing state. For example, the orientation of the traffic light may include one or more of the following properties: whether the traffic light is facing forward, not facing forward, facing backward, facing left, facing right, unknown, etc. Another example: The housing shape may be vertical, horizontal, like a doghouse, like a pedestrian hybrid light, unknown, etc. Active light colors may include red, yellow, green, white, orange, unknown, or any other color.Active light source shapes may in some cases include a circle, a straight arrow, a left arrow, a right arrow, a downward arrow, a U-turn arrow, a bus, a number (e.g., countdown timer), a tram, a train, a bicycle, a pedestrian, a hand, unknown, or any other shape.

[0022] In some cases, the machine learning models may also include a fusion head trained to predict a final classification of the traffic light represented in the input image. In some examples, the fusion head can predict the final classification of the traffic light based on the traffic light attributes determined by the component heads. Additionally or alternatively, the traffic light fusion head can classify based on a combined feature vector containing one or more features from the component-related feature vectors. In some examples, the final classification of the traffic light may include a combination of one or more of the recognized attributes.For example, the final classification may include or display a summary of the most important attributes of the traffic light, relating to the current state of the traffic light and / or the traffic rules transmitted by the traffic light.

[0023] In some examples, the system(s) of this disclosure can, based on the traffic light attributes and / or classification determined by the machine learning model(s), cause a machine to perform one or more operations. For example, the system(s) of this disclosure can use the traffic light attributes and / or classification to determine the active traffic rules for a current lane segment in which the machine is operating. Based on the traffic rules, the system(s) can cause the machine to perform one or more operations in accordance with the traffic rules (e.g., stop, proceed, turn, yield, etc.).

[0024] As described herein, the system(s) can also train the machine learning model(s) to predict the properties and / or classifications of the traffic lights. In some examples, to reduce the number of false-positive activations in each component head and the fusion head, the system can apply implicit negative training targets for each negative example and assign negative examples to a uniform distribution output. In this way, the machine learning model can better distinguish valid examples from unknown or invalid examples.

[0025] In some cases, when a component head (e.g., alignment) is allowed only one valid output, the system can apply softmax activation. For example, the implicit negative training goal for a negative example might be a uniform distribution greater than zero. The sum of the uniform distribution need not necessarily equal one; it might need to be less than the confidence threshold applied at test time. In this way, the confidence threshold applied at test time can determine whether a test example is a negative or a positive example. If the maximum activation across all classes is below the threshold, the example can be a negative example for that component or fusion head. Otherwise, the example can be a positive example for that component or fusion head, and the maximum activation across all classes can become the predicted class.In some examples, cross-entropy loss or another single-label loss function (e.g., focal loss), label smoothing, and / or another regularization procedure can be applied during training.

[0026] If, however, a component head (e.g., color-shape) is allowed to have more than one valid output, the system can apply sigmoidal activation. For example, the implicit negative training goal for a negative example might be a uniform distribution with only zero values. If the maximum activation across all classes is below the confidence threshold applied at test time, the example can be a negative example for that component or fusion head. Otherwise, the example can be a positive example for that component or fusion head, and any class with a probability at or above the confidence threshold can become a predicted class. In some examples, binary cross-entropy losses or other multi-label loss functions (e.g., binary focus losses), label smoothing, and other regularization techniques can be applied during training.

[0027] In various cases, the system(s) of this disclosure can use data augmentation methods to selectively update one or more components and the fusion class when working with the multi-component model architecture. These data augmentation methods can include classical image processing techniques, style transfer techniques, or other methods. For example, the system(s) can perform data enrichment or data augmentation to generate training examples for stateless lights by using image processing techniques to turn off the red / green / yellow lights. That is, the system(s) can identify the pixels that exhibit a strong red / green / yellow color and intensity and then modify these pixels to resemble the surrounding pixels.For example, with red light, the system can use a threshold value in the HSV (Hue, Saturation, Value) color space to identify pixels with higher red values ​​and brightness, and then adjust the saturation to a low value and the brightness to match the surrounding pixels. In another example, the system can perform grayscale amplification to convert an image to grayscale, removing all color information, and change the label class to "stateless." Using such techniques, the system can force the neural network to learn the concept of color.

[0028] Additionally or alternatively, the system can use a style transfer extension to train a neural network to learn the mapping between red / green / yellow traffic lights and gray / stateless traffic lights, or use a neural network with a style transfer extension to switch all traffic lights to green. In another example, the system can use a style transfer extension to change the shape and / or other properties of the light sources. For example, the system can change the shape of one or more traffic light sources from one shape to another, such as from a circle to a left-pointing arrow. These techniques can also be applied to change other properties of a traffic light, such as the number of light sources, the housing shapes, and so on.

[0029] In some embodiments, the systems and methods described here can be performed in a simulation environment (e.g., NVIDIA DriveSIM, IsaacSIM, and / or IsaacGYM) using simulated data (e.g., simulated sensor data from simulated sensors of a virtual or simulated machine). For example, simulated input data (e.g., simulated images representing traffic lights) can be used to determine traffic light characteristics and / or classifications, and this information can be used to perform operations associated with the virtual machine within the simulation environment. These simulated operations can be used to test the performance of the underlying algorithms, systems, and / or processes before deployment in the real world. In some cases, the simulation can be used to generate synthetic training data, e.g.,Image data representing images of various traffic lights from the simulation. The synthetic training data (in addition to or as an alternative to real-world data) can then be used to train machine learning models (e.g., DNNs) to classify traffic lights and determine the traffic rules conveyed by different states of the traffic lights.

[0030] In any case, for example, when a simulation environment is used for testing, validation, training, etc., the simulation environment and / or the associated training data can be rendered or otherwise generated using one or more light transport algorithms—such as ray tracing and / or path tracing algorithms. In some embodiments, the simulation environment and / or one or more objects, features, or components thereof (e.g., traffic lights) can be generated or managed within a three-dimensional (3D) content collaboration platform (e.g., NVIDIA's OMNIVERSE) for industrial digitization, generative physical AI, and / or other use cases, applications, or services. For example, the content collaboration platform or system can be a system for using or developing Universal Scene Descriptor (USD) data (e.g., OpenUSD) to manage objects, features, scenes, etc.within a simulated environment, a digital environment, etc. The platform can include real-world physics simulation, for example, using NVIDIA's PhysX SDK to simulate real-world physics and physical interactions with simulations hosted by the platform. The platform can integrate OpenUSD, along with ray tracing / path tracing / light transport simulation (e.g., NVIDIA's RTX rendering technologies), into software tools and simulation workflows for building, training, deploying, or testing AI systems—for example, systems for testing, validating, training (e.g., machine learning models, neural networks, etc.), and / or other tasks related to automotive engineering, robotics, machinery, and / or other applications. In some examples, the simulation environment can include a digital twin of a real-world environment, such as...a digital twin of a specific road section, warehouse, data center, factory, airport, geographical area, marine area or other real-world environment in which autonomous or semi-autonomous machines can be operated.

[0031] In some embodiments, remote control of a vehicle or other machine can be achieved using a remote control system. For example, the systems and methods described here can be used to classify traffic lights to assist a remote operator in controlling—or providing waypoints or other control or navigation cues to—an autonomous or semi-autonomous machine through an environment.

[0032] In some examples, the machine learning models described here (e.g., deep neural networks, language models, LLMs, VLMs, multimodal language models, perceptual models, tracking models, fusion models, transformer models, diffusion models, pure encoder models, pure decoder models, encoder-decoder models, neural rendering field models (NERF), etc.) can be packaged as a microservice—for example, an inference microservice (e.g., NVIDIA NIMs)—which may contain a container (e.g., an operating system-level (OS) virtualization package) that may include an application programming interface (API), a server layer, a runtime layer, and / or a model "engine."For example, the inference microservice can include the container itself and the model(s) (e.g., weights and biases). In some cases, such as when the machine learning model(s) is / are small enough (e.g., has / have a sufficiently small number of parameters), the model(s) can be integrated into the container itself. In other cases—for example, when the model(s) is / are large—the model(s) can be hosted / stored in the cloud (e.g., in a data center) and / or hosted on-premises and / or at the edge (e.g., on a local server or computer device, but outside the container). In such implementations, the model(s) can be accessed via one or more APIs, such as REST APIs.Therefore, the machine learning model described here can, in some implementations, be deployed as an inference microservice to accelerate the deployment of one or more models in any cloud, data center, or edge computing system while ensuring data security. For example, the inference microservice can include one or more APIs, a pre-configured container for simplified deployment, and an optimized inference engine (e.g., built using standardized software for deploying and running AI models, such as...).the Triton Inference Server from NVIDIA) and / or one or more APIs for high-performance deep learning inference, which may include an inference runtime environment and model optimizations that provide low latency and high throughput for production applications—such as NVIDIA's TensorRT) and / or enterprise management data for telemetry (e.g., including identity, metrics, health checks, and / or monitoring). The machine learning models described here may be included as part of the microservice along with accelerated infrastructure that can be deployed with a single command and / or orchestrated and automatically scaled using a container orchestration system on accelerated infrastructure (e.g., from a single device to the size of a data center). As such, the inference microservice may include the machine learning model(s) (e.g.,The inference microservice (optimized for high-performance inference) includes inference runtime software for executing the machine learning model(s) and providing outputs / responses to inputs (e.g., user requests, prompts, etc.), and enterprise management software for providing health checks, identity verification, and / or other monitoring. In some embodiments, the inference microservice may include software to perform an on-premises replacement and / or update of the machine learning model(s). During the replacement or update, the software performing the replacement / update may retain the user configurations of the inference runtime software and the enterprise management software.

[0033] The systems and procedures described here can be used without restriction by non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (autonomous vehicles or machines), manned and unmanned robots or robotic platforms, warehouse vehicles, all-terrain vehicles, vehicles coupled with one or more trailers, aircraft, boats, shuttles, emergency vehicles, motorcycles, electric or motorized bicycles, airplanes, construction vehicles, underwater vehicles, drones, and / or other types of vehicles.Furthermore, the systems and methods described here can be used for a variety of purposes, including but not limited to machine control, machine locomotion, machine propulsion, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and monitoring, simulation and digital twins, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital twins, data center processing, conversational AI, light transport simulation (e.g., ray tracing, path tracking, etc.), collaborative content creation for 3D assets, cloud computing, and / or other suitable applications.

[0034] The disclosed embodiments can be included in a variety of different systems, such as automotive systems (e.g., a control system for an autonomous or semi-autonomous machine, a perception system for an autonomous or semi-autonomous machine), systems implemented using a robot, aerospace systems, medical systems, boat systems, intelligent area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using an edge device, and systems implementing language models, such as...Large language models (LLMs), image language models (VLMs) and / or multimodal language models, systems that implement one or more image language models (VLMs), systems that implement one or more multimodal language models, systems that use or employ one or more inference microservices, systems that implement one or more machine learning models in a service or microservice together with an operating system-level virtualization package (e.g.,systems that are implemented in a container), systems that integrate one or more virtual machines (VMs), systems for performing operations to generate synthetic data, systems that are at least partially implemented in a data center, systems for performing conversational AI operations, systems for performing light transport simulations, systems for performing collaborative content creation for 3D assets, systems for performing generative AI operations, systems that are at least partially implemented using cloud computing resources, and / or other types of systems.

[0035] Referring to Fig. 1 is Fig. Figure 1 is a data flow diagram illustrating an example of a process 100 for classifying traffic lights using a multi-component machine learning model according to some embodiments of the present disclosure. It is understood that this and other arrangements described herein are given only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, commands, function groups, etc.) may be used in addition to or instead of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional units that can be implemented as discrete or distributed components, or in conjunction with other components, in any suitable combination and at any suitable location.Various functions described herein as being performed by units can be performed by hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. In some embodiments, the systems, methods, and processes described herein can be implemented using similar components, features, and / or functions to those of the Autonomous Vehicle 1000 example from the [reference to be added]. Fig. 10A-10D, of the example of a device 1100 from Fig. 11 and / or the example of a data center 1200 from Fig. 12 will be executed.

[0036] The process 100 can be implemented using additional or alternative components, including one or more sensors 102, one or more machine learning models 104, which may comprise a common backbone 106 and a variety of component heads 108(1)-108(N) (hereinafter collectively referred to as "component heads 108"), a classifier 110, and one or more powertrain components 112. In brief, the process 100 can be described as follows: the sensor(s) 102 can generate sensor data 114 (e.g., image data representing an image of a traffic light) that is applied to the machine learning model(s) 104. The backbone 106 of the model(s) 104 can generate a variety of features (e.g., feature vectors) and / or embeddings that are fed into the various component heads 108.The component heads 108 can predict one or more traffic light properties 116, which can be sent to the drive stack component(s) 112. Additionally or alternatively, the traffic light attributes 116 can be used by the classifier 110 (which may be part of the machine learning model(s) 104 or separate from the model(s) 104) to determine one or more traffic light classifications 118, which can also be sent to the drive stack component(s) 112. The drive stack component(s) 112 can use the traffic light attribute(s) 116 and / or the traffic light classification(s) 118 to cause a machine (e.g., the vehicle 1000) to perform one or more operations.

[0037] In some examples, the sensors 102 may include one or more of the vehicle's sensors 1000. For example, the sensors 102 may include one or more LiDAR sensors, radar sensors, image sensors (e.g., cameras), ultrasonic sensors, or other sensors described herein. Therefore, the sensor data 114 may include LiDAR data generated using the LiDAR sensor(s), radar data generated using the radar sensor(s), image data generated using the image sensor(s), ultrasonic data generated using the ultrasonic sensor(s), or any other type of sensor data.

[0038] In some examples, the sensor data 114 can be acquired in one format (e.g., RCCB, RCCC, RBGC, etc.) and then converted to another format (e.g., during sensor data preprocessing). In some other examples, the sensor data 114 can be provided as input to a sensor data or image data preprocessor (not shown) to generate preprocessed image data. Many types of images or formats can be used as inputs, such as compressed images like those in the Joint Photographic Experts Group (JPEG), Red Green Blue (RGB), or Luminance / Chrominance (YUV) formats, or compressed images as frames from a compressed video format (e.g., BH264 / Advanced Video Coding (AVC), H.265 / High Efficiency Video Coding (HEVC), VP8, VP9, ​​Alliance for Open Media Video 1 (AV1), Versatile Video Coding (VVC) or another video compression standard), raw images, for example from Red Clear Blue (RCCB), Red Clear (RCCC) or another image sensor type. In some examples, different formats and / or resolutions could be used for training the machine learning model(s) 104 than for inference (e.g. during the use of the machine learning model(s) 104 in the vehicle 1000).

[0039] A sensor or image data preprocessor can use data representative of one or more images (or other data representations, such as LiDAR depth maps) and load the sensor data into memory as a multidimensional array / matrix (in some examples also referred to as a tensor, or more precisely, an input tensor). The array size can be calculated and / or represented as W × H × C, where W is the image width in pixels, H is the height in pixels, and C is the number of color channels. Without loss of generality, other types and arrangements of input image components are also possible. In some embodiments, batching can be used for training and / or inference. In such examples, the batch size B can be used as a dimension (e.g., an additional fourth dimension). Thus, the input tensor can represent an array of dimension W × H × C × B.Any order of dimensions is possible, depending on the specific hardware and software used to implement the sensor data or image data preprocessor. This order can be chosen to maximize the training and / or inference performance of the machine learning model(s).

[0040] In some embodiments, an image preprocessing pipeline can be used by the sensor or image data preprocessor to acquire one or more data points from the sensor (e.g., camera(s)) and contained in the sensor data 114 to generate preprocessed image or sensor data that can represent an input image or images for the input layer(s) (e.g., backbone 106 and / or feature extraction layers) of the machine learning model(s) 104. An example of a suitable image preprocessing pipeline can take a raw RCCB Bayer type image (e.g., 1-channel) from the sensor and convert this image into an RCB planar image (e.g., 3-channel) stored in a fixed-precision format (e.g., 16 bits per channel). The image preprocessing pipeline may include decompression, noise reduction, demosaicing, white balance, histogram calculation and / or adaptive global tone mapping (e.g., in this order or in an alternative order).

[0041] If the image data preprocessor performs noise reduction, this may include bilateral noise reduction in the Bayer region. If the image data preprocessor performs demosaicing, this may include bilinear interpolation. If the sensor or image data preprocessor performs histogram calculation, this may include calculating a histogram for the C-channel and, in some cases, combined with decompression or noise reduction. If the sensor or image data preprocessor uses adaptive global tone mapping, this may include performing an adaptive gamma log transformation. This may include calculating a histogram, determining a midtone value, and / or estimating a maximum luminance using the midtone value.

[0042] In some examples, the sensor or image data preprocessor can create a section of a traffic light from a larger image. For example, Fig. 2. A representation of an image section 202 showing a traffic light 204, wherein the image section 202 is obtained from image data representing an image 206 of an environment 208, according to some embodiments of the present disclosure. In some cases, the sensor or image data preprocessor can perform object detection or mapping to identify and locate the traffic lights 204 within the image 206. That is, the sensor or image data preprocessor can determine which pixels in the image 206 correspond to the traffic light 204 and / or a buffer area around the traffic light 204, as shown in the image section 202. The sensor or image data preprocessor can then generate the image section 202 based on the identification of these pixels or otherwise by locating the traffic light 204 within the image 206.

[0043] As described herein, the sensor data 114 (e.g., the image section 202, the preprocessed sensor data or image data, etc.) can be used as input for the machine learning model(s) 104. In some examples, the machine learning model(s) 104 may include one or more deep neural networks (DNNs). Although examples relating to the use of neural networks, and in particular DNNs, as machine learning model(s) 104 are described here, this is not intended to be a limitation. For example, and without restriction, the models for machine learning 104 can include any type of machine learning model, such as machine learning models that employ linear regression, logistic regression, decision trees, support vector machines (SVMs), Naive Bayes, k-nearest neighbors (Knn), K-means clustering, random forest, dimensionality reduction algorithms, gradient enhancement algorithms, neural networks (e.g.Autoencoders, convolutional networks, recurrent networks, perceptrons, Long / Short Term Memory (LSTM), Hopfield, Boltzmann, Deep Belief, deconvolutional networks, generative adversarial networks, Liquid State Machines, etc.) and / or other types of machine learning models.

[0044] As in the example in Fig. As shown in Figure 1, the machine learning model (or models) 104 may, in some cases, comprise a multi-component architecture. That is, the machine learning model (or models) 104 may include a common backbone 106 and a variety of component heads 108. The backbone 106 can serve as the basic architecture of the machine learning model (or models) 104. For example, the backbone 106 may comprise a number of layers and / or components that process the input data to extract meaningful features. In some examples, the backbone 106 may comprise one or more trained neural networks optimized for tasks such as image classification or feature extraction.The backbone 106 can transform raw data into a higher-dimensional representation, capturing essential patterns and structures that facilitate the machine learning model(s) 104's ability to learn and make predictions. By utilizing the features learned from the backbone 106, additional layers of the machine learning model(s) 104—such as the component heads 108 and / or other layers or components—can fine-tune the model(s) 104 for specific tasks, thereby improving accuracy and performance.

[0045] In various examples, each of the component heads 108(1)-108(N) (where N can represent any number of component heads) can be trained or optimized to recognize one or more attributes of one or more traffic lights represented in an input. For example, the first component head 108(1) of the machine learning model(s) 104 can be trained to predict one or more first attributes of the traffic light (e.g., an orientation of the traffic light), while a second component head (not shown) of the machine learning model(s) 104 can be trained to predict one or more second attributes of the traffic light (e.g., housing shape(s)), an Nth component head 108(N) of the machine learning model(s) 104 can be trained to predict one or more Nth attributes of the traffic light (e.g., active light color(s) and / or shape(s)), and so on.In various examples, a single component head can have more than one valid output for a given input. For instance, a component head designed to predict active light sources can output a message indicating that the active light sources of a traffic light include a round light and an arrow light if the round light and arrow light of the traffic light are illuminated simultaneously (e.g., a green circle and a green left-pointing arrow).

[0046] In various examples, the machine learning models 104 can use the component heads 108 to output the traffic light attributes 116 associated with the traffic light device from the input. For example, each of the component heads 108 can generate a respective probability vector for the attributes on which each component head is trained to recognize. For example, if the first component head 108(1) is trained to recognize active light source color attributes, the first component head 108(1) can generate a probability vector containing entries that indicate the probability or reliability of the active light source colors for the traffic light. For example, if the active light source color of the traffic light is green, the entries in the probability vector for the active light source color red or yellow might have a low value (e.g.,1%), while the entry for the active light source color green can have a higher value (e.g. 98%).

[0047] In some examples, the traffic light attributes 116 can include, among other things, the orientation, the housing shape, the color of the active light source, the shape of the active light source, the number of light sources, obstruction (e.g., whether the light is obscured by other objects), clipping (e.g., where the light is cut off by the image edge), the road user (e.g., for which type(s) of road users the traffic light applies), and / or the flashing state. For example, the orientation of the traffic light can include one or more of the following properties: whether the traffic light is facing forward, not facing forward, facing backward, facing left, facing right, unknown, etc. Another example: The housing shape can be vertical, horizontal, like a doghouse, like a pedestrian hybrid lamp, unknown, etc.Active light source colors can include red, yellow, green, white, orange, unknown, or any other color. Active light source shapes can, in some cases, include a circle, a straight arrow, a left arrow, a right arrow, a downward arrow, a U-turn arrow, a bus, a number (e.g., a countdown timer), a tram, a train, a bicycle, a pedestrian, a hand, unknown, or any other shape. These are only some examples of attributes, which are not intended to be restrictive, and traffic light attributes 116 may include other attributes in additional or alternative examples.

[0048] In some examples, the classifier 110 can use the traffic light attributes 116 and / or other input data (not shown) to generate the traffic light classifications 118. Although in the example in Fig. While classifier 110 is represented as a separate component of the machine learning model(s) 104, in some examples it can be a component of the machine learning model(s) 104. For example, in the example described here, classifier 110 can be part of the fusion head 414 of the machine learning model(s) 104. Fig. 4. In some examples, the classifier 110 can predict the traffic light classification(s) 118 based on at least one or more of the traffic light attributes 116. For example, if the traffic light attributes indicate that the traffic light housing is vertical, the total number of lamps is three, and the active lamp color is green, the classifier 110 can classify the traffic light as a green light. In some examples, the traffic light classification(s) 118 may include a combination of one or more of the detected traffic light attributes 116. For example, the traffic light classification(s) 118 may include or indicate a summary of the most important attributes of the traffic light relating to the current state of the traffic light and / or the traffic rules transmitted by the traffic light.

[0049] For example, it shows Fig. Three different examples of traffic light configurations and states that can be classified using a multi-component machine learning model according to some embodiments of the present disclosure. Each of the examples in the Fig. The traffic lights shown in Figures 302-316 represent different states of traffic light vehicles that may occur in an environment and can be classified using the techniques of the present disclosure.

[0050] For example, traffic light 302 shows a vertical traffic light that is directed forward, has a total of three light sources, is not obscured or cut off, and has an active light source color in the form of a red circle. In such an example, the traffic light attributes 116 can specify that traffic light 302 has a vertical housing, is directed forward, has three light sources, and the active light source color and shape is a red circle. Additionally, the traffic light classifications 118 can specify that the class of traffic light 302 is a red circle. In contrast, traffic light 304 shows a vertical traffic light that is directed forward, has a total of three light sources, is not obscured or cut off, and has an active light source color of green in the form of a circle.In such an example, the traffic light attribute 116 can indicate that the traffic light 304 has a vertical housing, is directed forward, has three light sources, and the active light source color and shape is a green circle. Additionally, the traffic light classification 118 can indicate that the class of traffic lights 304 is at least a green circle.

[0051] Traffic light 306 is a vertical traffic light directed forward, has a total of three light sources, is not obscured or cut off, and is stateless (e.g., no active light sources). For traffic light 306, traffic light attribute 116 can indicate that the traffic light 306 has a vertical housing, is directed forward, has three light sources, and is stateless or unknown. Additionally, traffic light classification 118 can indicate that the class of traffic light 306 is stateless. Traffic light 308 comprises a horizontal housing, is directed forward, has a total of three light sources, is not obscured or cut off, and has an active light source color of yellow in the form of a circle.In such an example, the traffic light attribute 116 can indicate that the traffic light 308 has a horizontal housing, is directed forward, has three light sources, and the active light source color and shape is a yellow circle. Additionally, the traffic light classification 118 can indicate that the class of the traffic light 308 is at least a yellow circle.

[0052] Traffic light 310 is a vertical traffic light directed forward, with a total of three light sources, which is neither obscured nor cut off, and has an active light source color of green in the form of a left-pointing arrow. For traffic light 310, traffic light attribute 116 can indicate that the traffic light 310 has a vertical housing, is directed forward, has three light sources, and has a green left-pointing arrow as its active light source color / shape attribute. Additionally, traffic light classification 118 can indicate that the class of traffic light 310 is a green left-pointing arrow. Traffic light 312 is a vertical traffic light directed forward, with a total of four light sources, which is neither obscured nor cut off, and has an active light source color in the form of a red circle.For traffic light 312, the traffic light attributes 116 can indicate that the traffic light 312 has a vertical housing, is directed forward, has four light sources, and has a red circle as its active light source color-shape attribute. Additionally, the traffic light classifications 118 can indicate that the class of traffic light 312 is at least one red circle.

[0053] Traffic light 314 is a forward-facing traffic light with five lamps, which is neither obscured nor cut off and has active lamps in the colors green, a green left-pointing arrow, and a green circle. For traffic light 314, traffic light attributes 116 can specify that the traffic light 314 has a doghouse housing, is forward-facing, has five lamps, and has a green left-pointing arrow and a green circle as active lamp color-shape attributes. Additionally, traffic light classifications 118 can specify that the class of traffic light 314 is a green left-pointing arrow and a green circle. Finally, traffic light 316 is a vertical traffic light, which is forward-facing, has a total of three lamps, is neither obscured nor cut off, and has an active lamp color of yellow in the form of a circle.For traffic light 316, the traffic light attribute 116 can indicate that the traffic light 316 has a vertical housing, is directed forward, has three light sources, and has a yellow circle as its active light source color and shape. Additionally, the traffic light classification 118 can indicate that the class of traffic light 316 is a yellow circle.

[0054] Referring to the example in Fig. 1. The traffic light attributes 116 and / or the traffic light classifications 118 can be passed to the drive stack components 112. In some examples, the drive stack component(s) 112 can cause a machine (e.g., the vehicle 1000) to perform one or more operations based at least on the traffic light attributes 116 and / or the traffic light classification(s) 118. For example, the drive stack component(s) 112 can use the traffic light attributes 116 and / or the traffic light classifications 118 to determine the active traffic rules for a current lane segment in which the machine is traveling. Based on the traffic rules, the drive stack component(s) 112 can cause the machine to perform one or more operations according to the traffic rules (e.g., stop, proceed, turn, yield, etc.).

[0055] In various examples, the powertrain component(s) 112 may include a perception component, a modeling component, a planning component, a control component, an evasive component, an actuation component, a wait sensing component, and / or other components corresponding to additional and / or alternative layers of the powertrain. These components may use the traffic light attributes 116 and / or the traffic light classifications 118 as inputs to make various decisions related to control operations on behalf of the machine.

[0056] In some examples, one or more of the powertrain components 112 can assist the machine in avoiding collisions with objects (e.g., dynamic and stationary objects) and / or traffic violations. For example, one or more of the powertrain components 112 can use the traffic light attributes 116 and / or the traffic light classifications 118 to determine whether the machine's intended behavior or path would violate traffic rules (e.g., proceeding through a red light, stopping at a green light, turning when a turn signal is required, etc.). Furthermore, in some examples, one or more of the powertrain components 112 can use the traffic light attributes 116 and / or the traffic light classifications 118 to determine when the machine should cross an intersection, when it should stop at the intersection, when the machine has the right of way, etc.

[0057] Referring to Fig. 4 is Fig. 4 A data flow diagram illustrating an example of a process 400 for using a multi-component machine learning model architecture to predict attributes and / or classifications of traffic lights from image data according to some embodiments of the present disclosure. As shown, the image data 402—which may correspond to one or more of the sensor data 114, the image section 202, and / or the image 206—can be applied to the machine learning model(s) 420. In some examples, the machine learning models 420 can correspond to the machine learning models 104 in the example of Fig. 1. be similar to or identical with these. For example, the machine learning models 420 can comprise a multi-component deep neural network.

[0058] As shown, the machine learning model(s) 420 can include the common backbone 106. Based on the input image data 402, the backbone 106 can generate a common feature vector 404. The common feature vector 404 can include a numerical representation that captures one or more essential features or patterns of an image rendered using the image data 402. In some cases, the common feature vector 404 can consist of multiple values, each representing a specific feature, such as color, texture, edges, or shapes present in the image, or a combination of any of these features. For example, the backbone 106 can generate the common feature vector 404 to project the complex information of the image either into a lower-dimensional space or into a higher-dimensional space.

[0059] The common feature vector 404 can be used as input for a variety of component feature extractors 406(1)-406(N). The component feature extractors 406(1)-406(N) can analyze the common feature vector 404 to generate respective component feature vectors 408(1)-408(N), which can each be individually applied to the different component heads 108(1)-108(N). For example, a first component feature vector 408(1) can be applied to the first component head 108(1), a second component feature vector (not shown) can be applied to a second component head (not shown) of the model(s) 420, an Nth component feature vector 408(N) can be applied to an Nth component head 108(N) of the model(s) 420, and so on. In such examples, the features contained in the component feature vectors 408 can enable the component heads 108 to make accurate predictions.In other words, the component feature vectors 408 can contain selected features from the common feature vector 404 that can be used by the component heads 108 to predict the attributes they are each trained to identify, while features from the common feature vector 404 that cannot be used by the component heads 108 are excluded. For example, a component feature vector applied to a component head for predicting traffic light orientation can exclude features related to active light source colors, and vice versa. In some examples, the common feature vector 404 can be used as input for a combined feature extractor 412, which can generate a combined feature vector 410. Additionally or alternatively, the combined feature vector 410 can be generated from the various component feature vectors 408(1)-408(N).

[0060] Using the component feature vectors 408(1)-408(N), the component heads 108(1)-108(N) can generate respective attribute probability vectors 416(1)-416(N). Similarly, the fusion head 414 can generate a classification probability vector 418 using the combined feature vector 410. These probability vectors can contain multiple probability values ​​(e.g., confidence values) for multiple attributes and / or classifications. For example, entries in the probability vectors can contain values ​​indicating the probability that a particular traffic light has certain attributes and / or belongs to a particular class.

[0061] With reference to Fig. 5 is Fig. Figure 5 shows a data flow diagram illustrating an example method 500 for training one or more machine learning models 512 to predict traffic light classes and / or attributes according to some embodiments of the present disclosure. As shown, the machine learning model(s) 512 (which may correspond to the machine learning model(s) 104 and / or 420) can be trained using input data 502 (e.g., training inputs). The input data 502 may include images representing traffic lights. In some examples, the images of the input data 502 may be generated using an augmentation component 516 to modify image data 518.

[0062] For example, the augmentation component 516 can use classical image processing techniques, style transfer techniques, or other methods to perform data augmentation with respect to the image data 518. For example, the augmentation component 516 can perform data augmentation to generate training examples for stateless traffic lights by using image processing techniques to turn off the red / green / yellow traffic lights. That is, the augmentation component 516 can identify the pixels in the image data 518 that have a strong red / green / yellow color and intensity, and then modify these pixels to resemble the surrounding pixels.For example, with red light, the augmentation component 516 can use a threshold value in the HSV color space (Hue, Saturation, Value) to detect the pixels with higher red values ​​and brightness, and then adjust the saturation to a low value and the brightness to match the surrounding pixels. In another example, the augmentation component 516 can perform grayscale augmentation to convert an image to a grayscale image, so that no color information is available. Using such techniques, the training engine 508 can train the machine learning model(s) 512 (e.g., neural network) to learn the concept of color.

[0063] Additionally or alternatively, the augmentation component 516 can use style transfer augmentation, and the training engine 508 can train the machine learning model(s) 512 to learn the mapping between red / green / yellow traffic lights and gray / stateless traffic lights, or use a neural network for style transfer augmentation to turn all traffic lights green. As another example, the augmentation component 516 can use style transfer augmentation to change the shape and / or other properties of the light sources represented in the image data 518. For example, the augmentation component 516 can change the shape of one or more traffic light sources from one shape to another, such as from a circle to a left-pointing arrow.Furthermore, these techniques can also be used to change other properties of a traffic light, such as the number of light sources, the housing shapes, the orientation, etc.

[0064] The machine learning model(s) 512 can be trained using the training input data 502 and corresponding ground truth data 504 (which can also be generated by the augmentation component 516). In some examples, the ground truth data 504 can contain various data specifying valid attributes or classes of traffic light devices, including annotations, labels, masks, values ​​(e.g., confidence levels), and / or the like. In some embodiments, the ground truth data 504 can, for example, specify actual confidence levels associated with attributes and / or classes of traffic light devices from the input data 502.

[0065] The input data 502 and the ground truth data 504 can be included as part of a training dataset generated using the augmentation component 516. Furthermore, the ground truth data 504 in each example can be synthetically generated (e.g., generated from computer models or renderings), real-world generated (e.g., designed and generated from real-world data), machine-automated (e.g., using feature analysis and learning to extract features from data and then generate labels), human-annotated (e.g., a labeler or annotation expert defines the position of the labels), and / or a combination thereof.

[0066] The training engine 508 can use one or more loss functions that measure the loss (e.g., error) in the output data 510 generated by the machine learning model(s) 512 compared to the reference data 504 and / or the input data 502. In some examples, the training engine 508 can compare the output data 510 from the machine learning model(s) 512 with the input data 502 and update one or more parameters 506 of the machine learning model(s) 512, at least based on the comparison 514. That is, the training engine 508 can update / optimize one or more parameters 506 associated with the machine learning model(s) 512 to reduce the losses / differences between the output data 510 (e.g., predicted traffic light attributes or classes) and the reference data 504 (e.g., reference traffic light attributes or classes). Any type of loss function can be used, such as...Cross-entropy loss, mean squared error, mean absolute error, mean bias error, and / or other loss function types. In some examples, different outputs may have different loss functions. In such examples, the loss functions can be combined into an overall loss, and the overall loss can be used to train (e.g., update the parameters) the machine learning model(s). In each example, backward computations can be performed to recursively compute gradients of the loss function(s) with respect to training parameters. In some examples, weights and biases of the machine learning model(s) can be used to compute these gradients.

[0067] In some examples, the system of the present disclosure can apply implicit negative training targets for each negative example, and the training engine 508 can map negative examples to a uniform distribution output to reduce the number of false positive activations in each component head (e.g., the component heads 108) and / or in the fusion head (e.g., the classifier 110 and / or the fusion head 414). In this way, the machine learning model 512 can better distinguish valid examples from unknown or invalid examples.

[0068] In some cases, when a component head (e.g., alignment) is allowed only one valid output, the training engine 508 can train the machine learning model(s) 512 to apply a softmax activation. For example, the implicit negative training goal for a negative example might be a uniform distribution greater than zero. The sum of the uniform distribution need not necessarily equal one, but may need to be less than the confidence threshold applied at test time. In this way, the confidence threshold applied at test time can determine whether a test example is a negative or positive example. If the maximum activation across all classes is below the threshold, the example can be a negative example for that component or fusion head.Otherwise, the example may be a positive example for this component or fusion head, and the maximum activation across all classes may become the predicted class. In some examples, cross-entropy loss or another single-label loss function (e.g., focal loss), label smoothing, and / or another regularization method may be applied during training.

[0069] If, however, a component head (e.g., color-shape) is allowed to have more than one valid output, the training engine 508 can train the machine learning model(s) 512 to apply sigmoidal activation. For example, the implicit negative training goal for a negative example might be a uniform distribution with only zero values. If the maximum activation across all classes is below the confidence threshold applied at test time, the example might be a negative example for that component or fusion head. Otherwise, the example might be a positive example for that component or fusion head, and any class with a probability at or above the confidence threshold might become a predicted class. In some examples, binary cross-entropy losses or other multi-label loss functions (e.g.,binary focus loss), labeling smoothing and other regularization techniques are applied.

[0070] In some examples, the 512 machine learning models can be packaged as a microservice—for example, an inference microservice (e.g., NVIDIA NIMs)—which can contain a container (e.g., an operating system-level (OS) virtualization package) that may include an application programming interface (API) layer, a server layer, a runtime layer, and / or a model engine. For example, the inference microservice can include the container itself and the model (e.g., weights and biases). In some cases, such as when the 512 machine learning model(s) is / are small enough (e.g., has / have a sufficiently small number of parameters), the 512 model(s) can be integrated into the container itself.In some implementations, the machine learning models described here can be deployed as an inference microservice to accelerate model deployment to any cloud, data center, or edge computing system while ensuring data security. For example, the inference microservice may include one or more APIs, a pre-configured container for simplified deployment, an optimized inference engine (e.g., built using standardized software for deploying and running AI models, such as NVIDIA's Triton Inference Server), and / or one or more APIs for high-performance deep learning inference, which may include an inference runtime environment and model optimizations that provide low latency and high throughput for production applications (such as NVIDIA's TensorRT), and / or enterprise management data for telemetry (e.g.,including identity, metrics, health checks, and / or monitoring). The machine learning models 512 described here can be included as part of the microservice along with accelerated infrastructure that can be deployed with a single command and / or orchestrated and automatically scaled using a container orchestration system on accelerated infrastructure (e.g., from a single device to the size of a data center). As such, the inference microservice can include the machine learning model(s) 512 (e.g., optimized for high-performance inference), inference runtime software for executing the machine learning model(s) 512 and providing outputs / responses to inputs (e.g., user requests, prompts, etc.), and enterprise management software for providing health checks, identity, and other monitoring.In some implementations, the inference microservice may include software to perform an on-premises exchange and / or update of the machine learning models. During the exchange or update, the software performing the exchange / update may retain the user configurations of the inference runtime software and the enterprise management software.

[0071] Fig. Figure 6 illustrates an example of a system capable of performing one or more of the processes described herein according to some embodiments of the present disclosure. As shown, the system 602 (which may represent and / or include the exemplary computer devices 1100 and / or the exemplary data center 1200) may include one or more processors 604 (which may be similar to and / or include the CPUs 1106 and / or the GPUs 1108) and a memory 606 (which may be similar to and / or include the memory 1104). For example, the memory 606 may store one or more of the machine learning models 104 and / or the classifier 110. Additionally, the processor(s) 604 may execute one or more of the machine learning models 104 and / or the classifier 110 to perform one or more of the processes described herein.

[0072] For example, the system 602 can receive sensor data 114 from the sensor(s) 102 of a machine 608 (which may correspond to the vehicle 1000). The sensor data 114 can be applied to the machine learning model(s) 104, and the machine learning model(s) 104 can predict one or more attributes associated with one or more traffic lights represented in the sensor data 114. The classifier 110 can use the traffic light attribute(s) to determine one or more classifications of the traffic lights. The traffic light attributes / classifications 610 can then be sent to the powertrain component(s) 112 of the machine 608. The powertrain component(s) 112 can cause the machine 608 to perform one or more control operations based on the traffic light attributes / classifications 610.

[0073] With reference to the Fig. In sections 7 through 9, each of the procedure blocks 700, 800, and 900 described here comprises a computational process that can be executed using any combination of hardware, firmware, and / or software. For example, various procedures can be executed by a processor that carries out instructions stored in memory. The procedures can also be implemented as computer-readable instructions stored on a computer storage medium. The procedures can be provided, among other things, by a standalone application, a service, a hosted service (standalone or in combination with another hosted service), or a plug-in for another product. Furthermore, procedures 700, 800, and 900 can be illustrated by way of example with respect to… Fig. 1. However, these procedures can additionally or alternatively be performed by any system or combination of systems, including but not limited to those described herein.

[0074] Fig. Figure 7 is a flowchart illustrating an example of a method 700 for classifying traffic lights according to some embodiments of the present disclosure. Method 700 comprises, in block B702, applying image data, representing an image showing one or more traffic light devices, as input for one or more deep neural networks (DNNs). For example, the image data 402 can be used as input for the machine learning model(s) 420, which may be one or more DNNs. The image data 402 can represent an image (e.g., a section of an image) showing the traffic light device(s).

[0075] Procedure 700, in block B704, comprises the determination of one or more attributes associated with the traffic light device(s) using one or more component heads of the DNN(s). For example, component heads 108(1)-108(N) can be used to determine or generate attribute probability vectors 416(1)-416(N). The attribute probability vectors 416 can indicate the detected attributes associated with the device(s).

[0076] Procedure 700, in block B706, involves determining one or more classifications associated with the traffic light device(s) using a fusion head of the DNN(s) and based on at least the attribute(s). For example, the fusion head 414 of the machine learning model(s) 420 can be used to determine the classification probability vector 418, which can indicate the classification(s) associated with the traffic light device(s). In some examples, the fusion head 414 can determine the classifications based on the combined feature vector 410, based on embeddings, and / or based on the attributes predicted by the component heads 108.

[0077] Method 700, in block B708, comprises causing a machine to perform one or more control operations based on at least one of the attributes or classification(s) associated with the traffic light device(s). For example, the powertrain component(s) 112 can cause the machine (e.g., the vehicle 1000) to perform the control operation(s) based at least on the attributes (e.g., the traffic light attribute(s) 116 and / or attribute probability vectors 416) or the classification(s) (e.g., traffic light classification(s) 118 and / or classification probability vector 418) associated with the traffic light devices.

[0078] Fig. Figure 8 is a flowchart illustrating an example of a method 800 for determining traffic light attributes using a multi-component machine learning architecture according to some embodiments of the present disclosure. Method 800 comprises, in block B802, determining one or more first attributes associated with a traffic light device depicted in an image, using a first component head of a machine learning model. For example, the machine learning model 104 can use the first component head 108(1) to determine the first attribute or attributes associated with the traffic light device.

[0079] Method 800, in block B804, comprises determining one or more second attributes associated with the traffic light device depicted in the image, using one or more second components of the machine learning model. For example, the machine learning model 104 can use the Nth component 108(N) to determine the second attribute or attributes associated with the traffic light device. In some examples, the second attributes may differ from the first attributes. For example, the first attributes may include a housing shape of the traffic light, and the second attributes may include a color and a shape of an active light source of the traffic light.

[0080] Method 800, as defined in block B806, comprises the execution of one or more operations associated with a machine, based on at least one of the first attributes or the second attributes. For example, the powertrain component(s) 112 can cause the machine (e.g., the vehicle 1000) to perform the operation(s) based on at least one of the first attributes or the second attributes. In some cases, the operations may include, among others, stopping the machine, accelerating the machine, setting a steering angle of the machine, determining whether a current operation of the machine violates one or more traffic rules, determining active traffic rules based on the attributes, and so on.

[0081] Fig. Figure 9 is a flowchart illustrating an example of Method 900 for training a multi-component machine learning model to predict traffic light attributes and / or classes according to some embodiments of the present disclosure. Method 900 comprises, in block B902, capturing an image representing a traffic light device with one or more first attributes. The image may, for example, be an original image representing an environment and / or the traffic light device within that environment. In some cases, the image may be captured by a camera of a machine operating in the environment.

[0082] Method 900, in block B904, includes updating one or more parts of the image to produce an updated image depicting the traffic light device with one or more second attributes. For example, the augmentation component 516 can update the part or parts of the image to produce the updated image depicting the traffic light device with the second attribute(s). In some examples, the second attribute(s) may differ from the first attribute(s). As a first example, the traffic light device may have an active red light source in the original image, while in the updated image it may have an active green light source, and the red light source may be inactive.Additionally or alternatively, the traffic light device in the original image can have a vertical housing, while the traffic light in the updated image can have a horizontal or doghouse-shaped housing. In each example, any number of different attributes of the traffic light can be updated or changed between the original and updated images.

[0083] Method 900, in block B906, comprises obtaining, based at least on applying the updated image as training input to one or more machine learning models comprising one or more heads, one or more outputs determined using the head(s). For example, the various component heads 108, based at least on applying the updated image as training input to the machine learning model(s) 512, can predict the attributes of the traffic light depicted in the updated image.

[0084] Procedure 900, in block B908, involves evaluating one or more differences between the output data and the reference data. For example, the training engine 508 can evaluate the difference(s) between the output data 510 and the ground truth data 504. To do this, the training engine 508 can compare the confidence values ​​contained in probability vectors with the actual values ​​from the ground truth. In some cases, losses between the output data 510 and the ground truth data 504 can be calculated to evaluate the difference(s).

[0085] Procedure 900, in block B910, involves updating one or more parameters associated with one or more heads of the machine learning model, at least based on the difference(s) between the output data and the reference data. For example, the training engine 508 can update the parameters 506 of the machine learning model(s) 512 to reduce the difference(s) (e.g., losses) between the output data 510 and the ground truth data 504. In some examples, the parameter update may include updating weights or biases of the model and / or one or more of the component heads.

[0086] The systems and methods described here can be used for a wide variety of purposes, including but not limited to machine control, machine motion, machine drive, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and monitoring, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, data center processing, conversational AI, light transport simulation (e.g., ray tracing, path tracking, etc.), collaborative content creation for 3D assets, cloud computing, generative AI, (large) language models, and / or other suitable applications.

[0087] The disclosed embodiments can be included in a variety of different systems, such as automotive systems (e.g., a control system for an autonomous or semi-autonomous machine, a perception system for an autonomous or semi-autonomous machine), systems implemented using a robot, flight systems, medical systems, boat systems, intelligent area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems implemented using an edge device, systems containing one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in a data center, systems for performing conversational AI operations, and systems for performing light transport simulations.Systems for the collaborative creation of content for 3D assets, systems that are implemented at least partially using cloud computing resources, systems for performing generative AI operations, systems that implement a large language model (LLM) or are used to perform operations using such a model, and / or other types of systems. EXAMPLE OF AN AUTONOMOUS VEHICLE

[0088] Fig. Figure 10A is a representation of an example of an autonomous vehicle 1000 according to some embodiments of the present disclosure. The autonomous vehicle 1000 (hereinafter referred to alternatively as "vehicle 1000") can, without limitation, be a personal motor vehicle such as a car, truck, bus, first responder vehicle, shuttle, electric or motorcycle, fire engine, police vehicle, ambulance, boat, construction vehicle, underwater vehicle, robotic vehicle, drone, airplane, a vehicle coupled with a trailer (e.g., a semi-trailer truck for transporting cargo), and / or another vehicle (e.g., an unmanned vehicle and / or a vehicle capable of carrying one or more passengers).Autonomous vehicles are generally described using levels of automation established by the National Highway Traffic Safety Administration (NHTSA), a division of the U.S. Department of Transportation, and the Society of Automotive Engineers (SAE) in "Taxonomy and Definitions for Terms Relating to Driving Systems for Motor Vehicles on the Road" (Standard No. J3016-201806, published June 15, 2018, Standard No. J3016-201609, published September 30, 2016, and earlier and future versions of this standard). The Vehicle 1000 can perform functions according to one or more of Levels 3 through 5 of the autonomous driving levels. The Vehicle 1000 can perform functions according to one or more of Levels 1 through 5 of the autonomous driving levels.For example, depending on its embodiment, the vehicle 1000 may be capable of providing driver assistance (Level 1), partial automation (Level 2), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5). The term "autonomous," as used here, may encompass all types of autonomy for the vehicle 1000 or any other machine, such as fully autonomous, highly autonomous, conditionally autonomous, partially autonomous, assistive autonomy, semi-autonomous, primary autonomous, or other designations.

[0089] The vehicle 1000 can include components such as a chassis, a vehicle body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other vehicle components. The vehicle 1000 can include a drive system 1050, such as an internal combustion engine, a hybrid-electric drive, a fully electric motor, and / or another type of drive system. The drive system 1050 can be connected to a powertrain of the vehicle 1000, which may include a transmission to enable the propulsion of the vehicle 1000. The drive system 1050 can be controlled in response to signals received from the throttle / accelerator pedal 1052.

[0090] A steering system 1054, which may include a steering wheel, can be used to steer the vehicle 1000 (e.g., along a desired path or route) when the drive system 1050 is in operation (e.g., when the vehicle is in motion). The steering system 1054 can receive signals from a steering actuator 1056. The steering wheel may be optional for full automation functionality (Level 5).

[0091] The brake sensor system 1046 can be used to actuate the vehicle brakes in response to receiving signals from the brake actuators 1048 and / or brake sensors.

[0092] The controller(s) 1036, which includes one or more system-on-chips (SoCs) 1004 ( Fig. The controller(s) 1036 may include one or more onboard (e.g., integrated) computing units (e.g., GPUs) that process sensor signals and issue operating commands (e.g., command-representing signals) to one or more components and / or systems of the vehicle 1000. For example, the controller(s) may send signals to actuate the vehicle brakes via one or more brake actuators 1048, to actuate the steering system 1054 via one or more steering actuators 1056, and to actuate the propulsion system 1050 via one or more throttle / accelerator actuators 1052. The controller(s) 1036 may include one or more onboard (e.g., integrated) computing units (e.g., supercomputers) that process sensor signals and issue operating commands (e.g., command-representing signals) to enable autonomous driving and / or to assist a human driver in driving the vehicle 1000.The controller(s) 1036 can include a first controller 1036 for autonomous driving functions, a second controller 1036 for functional safety functions, a third controller 1036 for artificial intelligence functions (e.g., computer vision), a fourth controller 1036 for infotainment functions, a fifth controller 1036 for emergency redundancy, and / or other controllers. In some examples, a single controller 1036 can perform two or more of the above-mentioned functions, two or more controllers 1036 can perform a single function, and / or any combination thereof.

[0093] The controller(s) 1036 can provide the signals for controlling one or more components and / or systems of the vehicle 1000 in response to sensor data received from one or more sensors (e.g. sensor inputs). The sensor data can be used, for example, without restriction, from global navigation satellite systems (“GNSS”) sensors 1058 (e.g., Global Positioning System sensors), radar sensors 1060, ultrasonic sensors 1062, lidar sensors 1064, inertial measurement unit (IMU) sensors 1066 (e.g., accelerometers, gyroscopes, magnetic compasses, magnetometers, etc.), microphone(s) 1096, stereo camera(s) 1068, wide-angle camera(s) 1070 (e.g., fisheye cameras), infrared camera(s) 1072, surround camera(s) 1074 (e.g., 360-degree cameras), long-range and / or medium-range camera(s) 1098, speed sensor(s) 1044 (e.g., for measuring the speed of the vehicle 1000), vibration sensor(s) 1042, steering sensor(s) 1040, brake sensor(s) (e.g.as part of the brake sensor system 1046) and / or other sensor types.

[0094] One or more of the control units 1036 can receive inputs (e.g., represented by input data) from an instrument cluster 1032 of the vehicle 1000 and provide outputs (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display 1034, an acoustic signal generator, a loudspeaker, and / or via other components of the vehicle 1000. The outputs can provide information such as vehicle speed, time, map data (e.g., the high-resolution ("HD") map 1022 from Fig. 10C), location data (e.g., the location of vehicle 1000, e.g., on a map), direction, location of other vehicles (e.g., an occupancy grid), information about objects and the status of objects as perceived by the controller(s) 1036, etc. For example, the HMI display 1034 can show information about the presence of one or more objects (e.g., a road sign, a warning sign, a changing traffic light, etc.) and / or information about driving maneuvers that the vehicle has performed, is currently performing, or will perform (e.g., changing lanes now, taking exit 34B in two miles, etc.).

[0095] The vehicle 1000 further includes a network interface 1024, which can use one or more wireless antennas 1026 and / or modems to communicate over one or more networks. For example, the network interface 1024 can be capable of communicating via Long-Term Evolution (“LTE”), Wideband Code Division Multiple Access (“WCDMA”), Universal Mobile Telecommunications System (“UMTS”), Global System for Mobile Communication (“GSM”), IMT-CDMA Multi-Carrier (“CDMA2000”), etc. The wireless antenna(s) 1026 can also enable communication between objects in the environment (e.g., vehicles, mobile devices, etc.) via local area networks such as Bluetooth, Bluetooth Low Energy (“LE”), Z-Wave, ZigBee, etc., and / or Low Power Wide Area Networks (“LPWANs”) such as LoRaWAN, SigFox, etc.

[0096] Fig. 10B is an example of camera positions and fields of view for the autonomous vehicle 1000 from Fig. 10A according to some embodiments of the present disclosure. The cameras and their respective fields of view are an example of one embodiment and are not intended to be limiting. For example, additional and / or alternative cameras may be provided and / or the cameras may be arranged at different positions on the vehicle 1000.

[0097] The camera types can include, among others, digital cameras that can be adapted for use with the components and / or systems of the vehicle 1000. The camera(s) can operate at Automotive Safety Integrity Level (ASIL) B and / or another ASIL. Depending on the configuration, the camera types can have any desired frame rate, for example, 60 frames per second (fps), 120 fps, 240 fps, etc. The cameras can be equipped with rolling shutter, global shutter, other shutter types, or a combination thereof. In some examples, the color filter array may include a red-clear-clear-clear color filter array (RCCC), a red-clear-clear-blue color filter array (RCCB), a red-blue-green-clear color filter array (RBGC), a Foveon X3 color filter array, a Bayer sensor color filter array (RGGB), a monochrome sensor color filter array, and / or another type of color filter array.In some embodiments, clear pixel cameras, such as cameras with an RCCC, RCCB and / or RBGC color filter array, can be used to increase light sensitivity.

[0098] In some examples, one or more of the cameras can be used to perform advanced driver assistance systems (ADAS) functions (e.g., as part of a redundant or fail-safe design). For instance, a multifunctional monocular camera can be installed to provide features such as lane departure warning, traffic sign recognition, and intelligent headlight control. One or more of the cameras (e.g., all cameras) can simultaneously record and provide image data (e.g., video).

[0099] One or more cameras can be mounted in a bracket, such as a custom-made (three-dimensionally printed) bracket, to avoid stray light and reflections from inside the vehicle (e.g., reflections from the dashboard reflected in the windshield mirrors) that could impair the camera's image capture capabilities. Regarding the side mirror mounting brackets, these can be individually 3D printed so that the camera mounting plate matches the shape of the side mirror. In some cases, the cameras can be integrated into the side mirror itself. For side cameras, they can also be integrated into the four pillars at each corner of the cabin.

[0100] Cameras with a field of view that includes parts of the environment in front of the vehicle (e.g., forward-facing cameras) can be used for surround view to identify forward paths and obstacles and, with the aid of one or more controllers and / or control SoCs, provide information critical for creating an occupancy grid and / or determining preferred vehicle paths. Front-facing cameras can be used to perform many of the same ADAS functions as LiDAR, including emergency braking, pedestrian detection, and collision avoidance. Front-facing cameras can also be used for ADAS functions and systems, including lane departure warning (LDW), adaptive cruise control (ACC), and / or other functions such as traffic sign recognition.

[0101] In a forward-facing configuration, various cameras can be used, including, for example, a monocular camera platform with a complementary metal-oxide-semiconductor color imager (“CMOS”). Another example is a wide-angle camera (1070), which can be used to detect objects moving into the field of view from the periphery (e.g., pedestrians, crossing traffic, or bicycles). Although in Fig. Since only one wide-angle camera is shown in Figure 10B, there can be any number (including zero) of wide-angle cameras (1070) on the vehicle (1000). Furthermore, any number of long-range cameras 1098 (e.g., a pair of long-range stereo cameras) can be used for depth-based object detection, especially for objects for which a neural network has not yet been trained. The long-range cameras 1098 can also be used for object detection and classification, as well as for basic object tracking.

[0102] Any number of stereo cameras 1068 can also be included in a forward-facing configuration. In at least one embodiment, one or more of the stereo cameras 1068 can include an integrated control unit comprising a scalable processing unit that can provide programmable logic (“FPGA”) and a multi-core microprocessor with an integrated controller area network (“CAN”) or an Ethernet interface on a single chip. Such a unit can be used to create a 3D map of the vehicle's surroundings, including a distance estimate for all points in the image.An alternative stereo camera (stereo cameras) 1068 can comprise a compact stereo image sensor (stereo image sensors) that may include two camera lenses (one on each side, left and right) and an image processing chip that can measure the distance from the vehicle to the target object and use the generated information (e.g., metadata) to activate the autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 1068 can also be used in addition to or as an alternative to those described here.

[0103] Cameras with a field of view that includes parts of the surroundings to the sides of the vehicle (e.g., side cameras) can be used for all-around vision and provide information that is used to create and update the occupancy grid and to generate warnings of side-impact collisions. For example, surround cameras (e.g., four surround cameras, as in Fig. (shown in Figure 10B) are positioned on the vehicle 1000. The surround camera(s) 1074 can include wide-angle camera(s) 1070, fisheye camera(s), 360-degree camera(s), and / or the like. For example, four fisheye cameras can be positioned at the front, rear, and sides of the vehicle. In an alternative arrangement, the vehicle can use three surround cameras 1074 (e.g., left, right, and rear) and one or more other cameras (e.g., a forward-facing camera) as a fourth surround camera.

[0104] Cameras with a field of view that includes parts of the area behind the vehicle (e.g., reversing cameras) can be used for parking assistance, surround view, collision warning, and creating and updating the occupancy grid. A variety of cameras can be used, including but not limited to cameras that are also suitable as forward-facing cameras (e.g., long-range and / or medium-range cameras, stereo cameras, infrared cameras, etc.), as described herein.

[0105] Fig. 10C is a block diagram of an example system architecture for the example of an autonomous vehicle 1000 from Fig. 10A according to some embodiments of the present disclosure. It is understood that these and other arrangements described herein are given only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, instructions, functional groups, etc.) may be used in addition to or instead of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional units that may be implemented as discrete or distributed components, or in conjunction with other components, in any suitable combination and at any suitable location. Various functions described herein as being performed by units may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory.

[0106] Each of the vehicle's components, features, and systems 1000 in Fig. Vehicle 10C is shown connected via a bus 1002. The bus 1002 may include a Controller Area Network (CAN) data interface (referred to herein alternatively as the "CAN bus"). A CAN bus may be a network within the vehicle 1000 that supports the control of various features and functions of the vehicle 1000, such as the operation of brakes, acceleration, braking, steering, windshield wipers, etc. A CAN bus may be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., a CAN ID). The CAN bus can be read to determine the steering wheel angle, vehicle speed, engine revolutions per minute (RPM), button positions, and / or other vehicle status indicators. The CAN bus may be ASIL B compliant.

[0107] Although bus 1002 is described here as a CAN bus, this is not intended to be a limitation. For example, FlexRay and / or Ethernet can be used in addition to or as an alternative to the CAN bus. Furthermore, the representation of bus 1002 by a single line is not intended to be a limitation. For example, there can be any number of buses 1002, including one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other buses using a different protocol. In some examples, two or more buses 1002 can be used to perform different functions and / or for redundancy. For example, a first bus 1002 can be used for collision avoidance and a second bus 1002 for actuation control.In each example, each bus 1002 can communicate with each of the vehicle 1000's components, and two or more buses 1002 can communicate with the same components. In some examples, each SoC 1004, each controller 1036, and / or each computer within the vehicle can have access to the same input data (e.g., inputs from vehicle 1000 sensors) and be connected to a common bus, such as the CAN bus.

[0108] The vehicle 1000 can include one or more controllers 1036, as shown here in relation to Fig. 10A are described. The controller(s) 1036 can be used for a variety of functions. The controller(s) 1036 can be coupled with various other components and systems of the vehicle 1000 and used for controlling the vehicle 1000, for the artificial intelligence of the vehicle 1000, for infotainment for the vehicle 1000, and / or the like.

[0109] The vehicle 1000 can comprise one or more system-on-a-chip (SoC) 1004. The SoC 1004 can include one or more CPUs 1006, one or more GPUs 1008, one or more processors 1010, one or more caches 1012, one or more accelerators 1014, one or more data storage devices 1016, and / or other components and features not shown. The SoC(s) 1004 can be used to control the vehicle 1000 in a variety of platforms and systems. For example, the SoC(s) 1004 in a system (e.g., the system of the vehicle 1000) can be combined with an HD card 1022, which is accessed via a network interface 1024 from one or more servers (e.g., server(s) 1078). Fig. 10D) map updates and / or updates may be received.

[0110] The CPU(s) 1006 may comprise a CPU cluster or CPU complex (hereinafter referred to as "CCPLEX"). The CPU(s) 1006 may comprise multiple cores and / or L2 caches. In some embodiments, the CPU(s) 1006 may, for example, comprise eight cores in a coherent multiprocessor configuration. In some embodiments, the CPU(s) 1006 may comprise four dual-core clusters, each cluster having a dedicated L2 cache (e.g., a 2 MB L2 cache). The CPU(s) 1006 (e.g., the CCPLEX) may be configured to support simultaneous cluster operation, allowing any combination of CPU(s) 1006 clusters to be active at any given time.

[0111] The CPU(s) 1006 can implement power management features that include one or more of the following: individual hardware blocks can be automatically clocked when idle to dynamically conserve power; each core can be clocked when the core is not actively executing instructions due to the execution of WFI / WFE instructions; each core can be clocked independently; each core cluster can be clocked independently when all cores are clocked or power-locked; and / or each core cluster can be power-locked independently when all cores are power-locked. The CPU(s) 1006 can further implement an improved power state management algorithm, specifying acceptable power states and expected wake-up times, with the hardware / microcode determining the optimal power state for the core, cluster, and CCPLEX to transition to.The processor cores can support simplified sequences for switching to the performance state in the software, offloading the work to the microcode.

[0112] The GPU(s) 1008 may include an integrated GPU (referred to herein alternatively as the "iGPU"). The GPU(s) 1008 may be programmable and efficient for parallel workloads. The GPU(s) 1008 may, in some examples, use an extended Tensor instruction set. The GPU(s) 1008 may include one or more streaming microprocessors, each streaming microprocessor being able to include an L1 cache (e.g., an L1 cache with a minimum storage capacity of 96 KB), and two or more of the streaming microprocessors being able to share an L2 cache (e.g., an L2 cache with a storage capacity of 512 KB). In some embodiments, the GPU(s) 1008 may include at least eight streaming microprocessors. The GPU(s) 1008 can use one or more Compute Application Programming Interface(s) (API(s)). Furthermore, the GPU(s) 1008 can use one or more parallel computing platforms and / or programming models (e.g., ).B. NVIDIA CUDA).

[0113] The GPU(s) 1008 can be performance-optimized for optimal performance in automotive and embedded applications. For example, the GPU(s) 1008 can be manufactured using a FinFET field-effect transistor. However, this is not a limitation, and the GPU(s) 1008 can be manufactured using other semiconductor fabrication processes. Each streaming microprocessor can contain a number of mixed-precision processing cores divided into multiple blocks. For example, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA TENSOR COREs for deep learning matrix arithmetic, an L0 instruction cache, a warp scheduler, a dispatch unit and / or a 64 KB registry file.Furthermore, streaming microprocessors can include independent parallel integer and floating-point data paths to enable efficient execution of workloads with a mix of computations and address calculations. Streaming microprocessors can include independent thread scheduling capabilities to allow for finer synchronization and collaboration between parallel threads. Streaming microprocessors can also include a combined L1 data cache and a shared memory unit to improve performance while simplifying programming.

[0114] The GPU(s) 1008 can include High Bandwidth Memory (HBM) and / or a 16 GB HBM2 memory subsystem to provide a maximum memory bandwidth of approximately 900 GB / second in some examples. In some examples, synchronous graphics random access memory (SGRAM), such as Graphics Double Data Rate 5 (GDDR5) synchronous graphics random access memory, can be used in addition to or as an alternative to HBM memory.

[0115] The GPU(s) 1008 can incorporate a unified memory technology, including access counters, to enable more accurate migration of memory pages to the processor that accesses them most frequently, thereby improving efficiency for memory areas shared between processors. In some examples, support for Address Translation Services (ATS) can be used to allow the GPU(s) 1008 to directly access the page tables of the CPU(s) 1006. In such examples, if the memory management unit (MMU) of the GPU(s) 1008 encounters a mismatch, an address translation request can be passed to the CPU(s) 1006. In response, the CPU(s) 1006 can search its page tables for the virtual-to-physical mapping for the address and pass the translation back to the GPU(s) 1008.In this way, the unified memory technology can enable a single unified virtual address space for the memory of both the CPU(s) 1006 and the GPU(s) 1008, thereby simplifying the programming of the GPU(s) 1008 and the porting of applications to the GPU(s) 1008.

[0116] Furthermore, the GPU(s) 1008 may include an access counter that tracks the frequency of GPU(s) 1008 accessing the memory of other processors. This access counter can help ensure that memory pages are moved to the physical memory of the processor that accesses them most frequently.

[0117] The SoC(s) 1004 can include any number of Cache(s) 1012, including those described here. For example, the Cache(s) 1012 can include an L3 cache that is available to both the CPU(s) 1006 and the GPU(s) 1008 (i.e., connected to both the CPU(s) 1006 and the GPU(s) 1008). The Cache(s) 1012 can include a write-back cache capable of tracking the state of lines, for example, using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). The L3 cache can be 4 MB or larger, depending on the implementation, although smaller cache sizes are also possible.

[0118] The SoC(s) 1004 may include one or more ALU(s) that can be used to perform processing operations related to a variety of tasks or operations of the Vehicle 1000, such as DNN processing. Furthermore, the SoC(s) 1004 may include one or more Floating Point Units (FPUs)—or other types of mathematical or numeric coprocessors—for performing mathematical operations within the system. For example, the SoC(s) 1004 may include one or more FPUs integrated as execution units in one or more CPU(s) 1006 and / or GPU(s) 1008.

[0119] The SoC(s) 1004 can include one or more accelerators 1014 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, the SoC(s) 1004 can include a hardware acceleration cluster, which may include optimized hardware accelerators and / or a large amount of on-chip memory. The large on-chip memory (e.g., 4 MB SRAM) can enable the hardware acceleration cluster to accelerate neural networks and other computations. The hardware acceleration cluster can be used to complement the GPU(s) 1008 and offload some of the GPU(s) 1008's tasks (e.g., to free up more GPU 1008 cycles for other tasks). For example, the Accelerator 1014 can be used for targeted workloads (e.g., perception, Convolutional Neural Networks (CNNs), etc.) that are stable enough to be suitable for acceleration.The term “CNN”, as used here, can encompass all types of CNNs, including region-based or regional Convolutional Neural Networks (RCNNs) and Fast RCNNs (e.g., as used for object detection).

[0120] The Accelerator 1014 (e.g., the hardware acceleration cluster) can include one or more Deep Learning Accelerators (DLAs). The DLA can include one or more Tensor Processing Units (TPUs) that can be configured to provide an additional ten trillion operations per second for deep learning applications and inference. The TPUs can be accelerators configured and optimized to perform image processing functions (e.g., for CNNs, RCNNs, etc.). Furthermore, the DLA(s) can be optimized for a specific set of neural network types and floating-point operations, as well as for inference. The design of the DLA(s) can deliver more performance per millimeter than a general-purpose GPU and far surpasses the performance of a CPU.The TPU(s) can perform several functions, including single-instance folding, which supports, for example, the INT8, INT16 and FP16 data types for features and weights, as well as post-processing functions ().

[0121] The DLA(s) can run neural networks, especially CNNs, quickly and efficiently on processed or unprocessed data for a variety of functions, including, but not limited to: a CNN for object identification and recognition using data from camera sensors; a CNN for distance estimation using data from camera sensors; a CNN for the detection and identification of emergency vehicles using data from microphones; a CNN for facial recognition and vehicle owner identification using data from camera sensors; and / or a CNN for safety-related events.

[0122] The DLA(s) can execute any function of the GPU(s) 1008, and by using an inference accelerator, a developer can, for example, target either the DLA(s) or the GPU(s) 1008 for each function. For example, the developer can concentrate the processing of CNNs and floating-point operations on the DLA(s) and leave other functions to the GPU(s) 1008 and / or other accelerators 1014.

[0123] The Accelerator 1014 (e.g., the hardware acceleration cluster) can include a programmable image processing accelerator (PVA), which may alternatively be referred to here as a computer vision accelerator. The PVA(s) can be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. The PVA(s) can offer a balance between performance and flexibility. For example, each PVA can include any number of RISC (Reduced Instruction Set Computer) cores, DMA (Direct Memory Access), and / or any number of vector processors.

[0124] The RISC cores can interact with image sensors (e.g., the image sensors of one of the cameras described herein), image signal processors, and / or the like. Each RISC core can include any amount of memory. Depending on the implementation, the RISC cores can use one of several protocols. In some examples, the RISC cores can run a real-time operating system (RTOS). The RISC cores can be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, the RISC cores can include an instruction cache and / or tightly coupled RAM.

[0125] The DMA can allow components of the PVA(s) to access system memory independently of the CPU(s). The DMA can support any number of functions used to optimize the PVA(s), including, but not limited to, support for multidimensional addressing and / or circular addressing. In some examples, the DMA can support up to six or more addressing dimensions, including block width, block height, block depth, horizontal block spacing, vertical block spacing, and / or depth spacing.

[0126] Vector processors can be programmable processors designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing capabilities. In some examples, the PVA may comprise a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem may act as the primary processing engine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or vector memory (e.g., VMEM). A VPU core may include a digital signal processor, such as a one-instruction, multiple-data (SIMD), and very-long instruction word (VLIW) digital signal processor. The combination of SIMD and VLIW can improve throughput and speed.

[0127] Each vector processor can contain an instruction cache and be coupled to dedicated memory. Consequently, in some examples, each vector processor can be configured to operate independently of the others. In other examples, the vector processors contained in a particular PVA can be configured to utilize data parallelism. For example, in some embodiments, the multiple vector processors contained in a single PVA can execute the same computer vision algorithm, but on different regions of an image. In other examples, the vector processors contained in a particular PVA can simultaneously execute different computer vision algorithms on the same image, or even different algorithms on successive images or portions of an image.Among other things, any number of PVAs can be included in the hardware acceleration cluster, and any number of vector processors can be included in each of the PVAs. Furthermore, the PVA(s) can include additional error correction code (ECC) memory to improve the overall security of the system.

[0128] The Accelerator 1014 (e.g., the hardware acceleration cluster) may include an on-chip computer vision network and SRAM to provide high-bandwidth, low-latency SRAM for the Accelerator 1014. In some examples, the on-chip memory may comprise at least 4 MB of SRAM, consisting of, for example, and without limitation, eight field-configurable memory blocks accessible to both the PVA and the DLA. Each pair of memory blocks may include a memory, an APB (Advanced Peripheral Bus) interface, a configuration circuit, a controller, and a multiplexer. jAny type of memory can be used. The PVA and DLA can access the memory via a backbone that provides them with fast access. The backbone can include an on-chip computer vision network that connects the PVA and DLA to the memory (e.g., using the APB).

[0129] The on-chip computer vision network can include an interface that, before transmitting a control signal / address / data, determines that both the PVA and DLA provide readiness and validity signals. Such an interface can provide separate phases and channels for transmitting control signals / addresses / data, as well as burst communication for continuous data transmission. This type of interface can conform to ISO 26262 or IEC 61508 standards, but other standards and protocols can also be used.

[0130] In some examples, the SoC(s) 1004 may include a real-time ray-tracing hardware accelerator as described in US Patent Application No. 16 / 101,232, filed on August 10, 2018. The real-time ray-tracing hardware accelerator can be used to quickly and efficiently determine the positions and extents of objects (e.g., within a world model), to generate real-time visualization simulations, to interpret radar signals, to synthesize and / or analyze sound propagation, to simulate SONAR systems, for general wave propagation simulation, to compare with lidar data for localization and / or other functions, and / or for other uses. In some embodiments, one or more tree traversing units (TTUs) may be used to perform one or more ray-tracing-related operations.

[0131] The Accelerator 1014 (e.g., the hardware accelerator cluster) has a wide range of applications for autonomous driving. The PVA can be a programmable image accelerator used for key processing stages in ADAS and autonomous vehicles. The PVA's capabilities are well-suited for algorithmic domains requiring predictable processing with low power consumption and low latency. In other words, the PVA offers good performance for semi-dense or dense regular computations, even with small datasets, that demand predictable runtimes with low latency and low power consumption. In the context of autonomous vehicle platforms, the PVAs are therefore designed to execute classic computer vision algorithms, as they are efficient at object recognition and integer mathematics processing.

[0132] According to one embodiment of the technology, the PVA is used, for example, to perform computer stereo vision. In some examples, a semi-global, matching-based algorithm can be used, although this is not meant to be restrictive. Many Level 3 to 5 autonomous driving applications require real-time motion estimation / stereo matching (e.g., structure from motion, pedestrian detection, lane detection, etc.). The PVA can perform the computer stereo vision function using input from two monocular cameras.

[0133] In some examples, PVA can be used to perform dense optical flow processing. Process raw radar data (e.g., using a 4D Fast Fourier Transform) to provide processed radar data. In other examples, PVA is used for time-of-flight depth processing, for example, by processing raw time-of-flight data to provide processed time-of-flight data.

[0134] DLA can be used to power any type of network to improve control and driving safety, including, for example, a neural network that outputs a confidence score for each object detection. Such a confidence score can be interpreted as a probability or as the relative "weight" of each detection compared to other detections. This confidence score allows the system to make further decisions about which detections should be considered true positives rather than false positives. For example, the system can set a confidence threshold and consider only those detections that exceed the threshold as true positives. In an automatic emergency braking (AEB) system, false positives would cause the vehicle to automatically perform emergency braking, which is obviously undesirable.Therefore, only the most reliable detections should be considered as triggers for AEB. The DLA can operate a neural network to regress the reliability score. The neural network can use as input at least a subset of parameters, such as boundary frame dimensions, ground area estimates (e.g., obtained from another subsystem), the output signal of the inertial measurement unit (IMU) sensor 1066 correlated with the vehicle's orientation 1000, distance, and 3D position estimates of the object obtained from the neural network and / or other sensors (e.g., LIDAR sensor(s) 1064 or RADAR sensor(s) 1060), among others.

[0135] The SoC(s) 1004 may include data storage 1016 (e.g., memory). The data storage 1016 may be on-chip memory of the SoC(s) 1004 that can store neural networks to be executed on the GPU and / or the DLA. In some examples, the data storage 1016 may have a sufficiently large capacity to store multiple instances of neural networks for redundancy and security. The data storage 1012 may include L2 or L3 cache 1012. The reference to the data storage 1016 may include a reference to the memory associated with the PVA, the DLA, and / or other accelerators 1014, as described herein.

[0136] The SoC(s) 1004 can include one or more processors 1010 (e.g., embedded processors). The processor(s) 1010 can include a boot and power management processor, which may be a dedicated processor and a dedicated subsystem for handling boot power and management functions and the associated security enforcement. The boot and power management processor can be part of the boot sequence of the SoC(s) 1004 and provide runtime power management services. The boot and power management processor can provide clock and voltage programming, support for system power-saving transitions, management of the thermals and temperature sensors of the SoC(s) 1004, and / or management of the power states of the SoC(s) 1004.Each temperature sensor can be implemented as a ring oscillator whose output frequency is proportional to the temperature, and the SoC(s) 1004 can use the ring oscillators to detect the temperatures of the CPU(s) 1006, GPU(s) 1008, and / or accelerator(s) 1014. If it is detected that the temperatures exceed a threshold, the boot and power management processor can invoke a temperature fault routine and put the SoC(s) 1004 into a lower power state and / or put the vehicle 1000 into a chauffeur-to-safety-stop mode (e.g., bring the vehicle 1000 to a safe stop).

[0137] The 1010 processor(s) may further include a set of embedded processors that can serve as an audio processing engine. The audio processing engine can be an audio subsystem that provides full hardware support for multi-channel audio across multiple interfaces and a wide and flexible range of audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core with a digital signal processor and dedicated RAM.

[0138] The 1010 processor(s) may further include an always-on processor engine that can provide the necessary hardware functions to support low-power use cases for sensor management and wake-up. The always-on processor engine may include a processor core, tightly coupled RAM, supporting peripherals (such as timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0139] The 1010 processor(s) may also include a security cluster engine, which comprises a dedicated processor subsystem for managing automotive application security. The security cluster engine may include two or more processor cores, tightly coupled RAM, supporting peripherals (such as timers, an interrupt controller, etc.), and / or routing logic. In a security mode, the two or more cores may operate in lockstep mode, functioning as a single core with comparison logic to detect differences between their operations.

[0140] The 1010 processor(s) may further include a real-time camera engine, which may include a dedicated processor subsystem for managing real-time camera operations.

[0141] The 1010 processor(s) may further include a high dynamic range signal processor, which may include an image signal processor, which is a hardware engine that is part of the camera processing pipeline.

[0142] The 1010 processor(s) may include a video image compositor, which is a processing block (e.g., implemented on a microprocessor) that implements video post-processing functions required by a video playback application to generate the final image for the player window. The video image compositor may perform lens distortion correction for 1070 wide-angle cameras, 1074 surround-view cameras, and / or for indoor surveillance camera sensors. The indoor surveillance camera sensor is preferably monitored by a neural network running on a separate instance of the Advanced SoC, configured to identify indoor events and respond accordingly.A cabin system can perform lip reading to activate mobile service and make a call, dictate emails, change the vehicle's destination, activate or change the infotainment system and vehicle settings, or enable voice-controlled internet browsing. Certain functions are only available to the driver when the vehicle is operating in autonomous mode and are otherwise disabled.

[0143] The video image compositor can include enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, if motion occurs in a video, the noise reduction weights the spatial information accordingly and reduces the impact of information provided by neighboring frames. If a frame or part of a frame does not contain motion, the temporal noise reduction performed by the video image compositor can use information from the previous frame to reduce noise in the current frame.

[0144] The video image compositor can also be configured to perform stereo correction on input stereo lens frames. Furthermore, the video image compositor can be used for user interface composition when the operating system desktop is in use and the GPU(s) 1008 does not need to continuously render new surfaces. Even when the GPU(s) 1008 is powered on and actively performing 3D rendering, the video image compositor can be used to offload the GPU(s), thus improving performance and responsiveness.

[0145] The SoC(s) 1004 may further include a serial MIPI (Mobile Industry Processor Interface) camera interface for receiving video and camera input, a high-speed interface, and / or a video input block that can be used for camera and associated pixel input functions. The SoC(s) 1004 may also include one or more software-controlled input / output controllers that can be used to receive I / O signals not assigned to a specific role.

[0146] The SoC(s) 1004 can further include a wide range of peripheral interfaces to enable communication with peripheral devices, audio codecs, power management, and / or other devices. The SoC(s) 1004 can process data from cameras (e.g., connected via Gigabit Multimedia Serial Link and Ethernet), sensors (e.g., LiDAR sensor(s) 1064, RADAR sensor(s) 1060, etc., which can be connected via Ethernet), data from the 1002 bus (e.g., vehicle speed 1000, steering wheel position, etc.), and data from GNSS sensors 1058 (e.g., connected via Ethernet or CAN bus). The SoC(s) 1004 may also include dedicated high-performance mass storage controllers, which may contain their own DMA engines and which can be used to offload routine data management tasks from the CPU(s) 1006.

[0147] The SoC(s) 1004 can be an end-to-end platform with a flexible architecture that covers automation levels 3 to 5, thus offering a comprehensive functional safety architecture that leverages and efficiently utilizes computer vision and ADAS techniques for diversity and redundancy, and provides a platform for a flexible, reliable driving software stack and deep learning tools. The SoC(s) 1004 can be faster, more reliable, and even more energy- and space-efficient than conventional systems. For example, the 1014 accelerators, in combination with the 1006 CPUs, the 1008 GPUs, and the 1016 data storage devices, can provide a fast, efficient platform for autonomous vehicles of levels 3 to 5.

[0148] This technology thus offers capabilities and functions that cannot be achieved with conventional systems. For example, computer vision algorithms can be run on CPUs that can be configured with a higher-level programming language such as C to execute a variety of processing algorithms on a wide range of visual data. However, CPUs are often unable to meet the performance requirements of many computer vision applications, for example, in terms of execution time and power consumption. In particular, many CPUs are unable to execute complex object recognition algorithms in real time, which is a requirement for ADAS applications in vehicles and a prerequisite for practical Level 3 to 5 autonomous vehicles.

[0149] Unlike conventional systems, the technology described here, by providing a CPU complex, a GPU complex, and a hardware acceleration cluster, enables the simultaneous and / or sequential execution of multiple neural networks and the combination of their results to enable autonomous driving functions of levels 3 to 5. For example, a CNN running on the DLA or dGPU (e.g., the GPU(s) 1020) can include text and word recognition, allowing the supercomputer to read and understand traffic signs, including signs for which the neural network has not been specifically trained. The DLA can further include a neural network capable of identifying, interpreting, and semantically understanding the sign and passing this semantic understanding to the path planning modules running on the CPU complex.

[0150] As another example, multiple neural networks can be executed simultaneously, as is required for driving at levels 3, 4, or 5. For instance, a warning sign reading "Caution: Flashing lights indicate slippery conditions" along with an electric light can be interpreted independently or jointly by several neural networks. The traffic sign itself can be identified as a traffic sign by a first neural network (e.g., a trained neural network), while the text "Flashing lights indicate slippery conditions" can be interpreted by a second neural network, which informs the vehicle's path planning software (preferably running on the CPU) that the detection of flashing lights indicates slippery conditions.The flashing light can be identified by operating a third, deployed neural network over several frames, which informs the vehicle's route planning software about the presence (or absence) of flashing lights. All three neural networks can run simultaneously, for example, within the DLA and / or on the GPU(s) 1008.

[0151] In some examples, a CNN for facial recognition and vehicle owner identification can use data from camera sensors to identify the presence of an authorized driver and / or owner of the vehicle 1000. The always-on sensor processing engine can be used to unlock the vehicle when the owner approaches the driver's door and turns on the lights, and in security mode to disable the vehicle when the owner leaves. In this way, the SoC(s) 1004 provide security against theft and / or carjacking.

[0152] In another example, a CNN for emergency vehicle detection and identification can use data from microphones 1096 to detect and identify emergency vehicle sirens. Unlike conventional systems that use general classifiers for siren detection and manual feature extraction, the SoC(s) 1004 utilize the CNN to classify ambient and urban noise as well as visual data. In a preferred embodiment, the CNN running on the DLA is trained to identify the relative approach speed of the emergency vehicle (e.g., using the Doppler effect). The CNN can also be trained to identify emergency vehicles specific to the local area in which the vehicle operates, as identified by GNSS sensor(s) 1058.For example, if “ “ is operated in Europe, the CNN attempts to detect European sirens, and if it is operated in the United States, the CNN attempts to identify only North American sirens. Once an emergency vehicle is detected, a control program can be used to execute an emergency vehicle safety routine, in which the vehicle is slowed down, moved to the side of the road, parked, and / or held in neutral, using ultrasonic sensors 1062, until the emergency vehicle(s) has / have passed.

[0153] The vehicle may include one or more CPUs 1018 (e.g., discrete CPUs or dCPUs) that may be coupled to the SoC(s) 1004 via a high-speed connection (e.g., PCIe). The CPU(s) 1018 may, for example, include an x86 processor. The CPU(s) 1018 may be used to perform a variety of functions, including, for example, reconciling potentially inconsistent results between ADAS sensors and the SoC(s) 1004 and / or monitoring the status and functionality of the control unit(s) 1036 and / or the infotainment SoC 1030.

[0154] The Vehicle 1000 can include one or more GPUs 1020 (e.g., discrete GPUs or dGPUs) that can be coupled to the SoC(s) 1004 via a high-speed connection (e.g., NVIDIA's NVLINK). The GPU(s) 1020 can provide additional artificial intelligence functions, such as running redundant and / or different neural networks, and can be used to train and / or update neural networks based on inputs (e.g., sensor data) from the Vehicle 1000's sensors.

[0155] The vehicle 1000 may also include the network interface 1024, which may include one or more wireless antennas 1026 (e.g., one or more wireless antennas for different communication protocols, such as a cellular antenna, a Bluetooth antenna, etc.). The network interface 1024 can be used to establish a wireless connection via the internet to the cloud (e.g., to the server(s) 1078 and / or other network devices), to other vehicles, and / or to computer devices (e.g., passenger client devices). Communication with other vehicles can be established via a direct connection between the two vehicles and / or an indirect connection (e.g., via networks and the internet). Direct connections can be established via a vehicle-to-vehicle communication link.The vehicle-to-vehicle communication link can provide vehicle 1000 with information about vehicles in its vicinity (e.g., vehicles in front of, beside, and / or behind vehicle 1000). This functionality can be part of a cooperative adaptive cruise control function of vehicle 1000.

[0156] The network interface 1024 can include a system-on-a-chip (SoC) that provides modulation and demodulation functions, enabling the controller(s) 1036 to communicate over wireless networks. The network interface 1024 can include a high-frequency (RF) front end for up-conversion from baseband frequency to RF and down-conversion from RF to baseband frequency. The frequency conversions can be performed using known methods and / or superheterodyne techniques. In some examples, the RF front-end functionality can be provided by a separate chip. The network interface can include radio functionality for communication via LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other radio protocols.

[0157] The vehicle 1000 may further comprise one or more data storage devices 1028, which may include off-chip memory (e.g., outside the SoC(s) 1004). The data storage device(s) 1028 may comprise one or more memory elements, including RAM, SRAM, DRAM, VRAM, flash, hard disks, and / or other components and / or devices capable of storing at least one data bit.

[0158] The vehicle 1000 can also include GNSS sensors 1058. The GNSS sensors 1058 (e.g., GPS, supported GPS sensors, differential GPS sensors (DGPS), etc.) are used to support functions such as mapping, perception, occupancy grid generation, and / or path planning. Any number of GNSS sensors 1058 can be used, including, for example, and without limitation, a GPS unit that uses a USB connection with an Ethernet-to-serial (RS-232) bridge.

[0159] The vehicle 1000 may also include RADAR sensors 1060. The RADAR sensors 1060 can be used by the vehicle 1000 for remote vehicle detection, even in darkness and / or adverse weather conditions. The functional safety level of the RADAR can be ASIL B. The RADAR sensor(s) 1060 can use CAN and / or bus 1002 (e.g., for transmitting data generated by the RADAR sensor(s) 1060) for control and access to object tracking data, with some examples using Ethernet for accessing raw data. A variety of RADAR sensor types can be used. For example, and without limitation, the RADAR sensors 1060 can be suitable for use as front, rear, and side radar. Pulse Doppler RADAR sensors are used in some examples.

[0160] The 1060 RADAR sensor(s) can include various configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range with side coverage, etc. In some examples, a long-range RADAR can be used for adaptive cruise control. Long-range RADAR systems can provide a wide field of view, achieved through two or more independent scans, for example, within a range of 250 m. The 1060 RADAR sensor(s) can assist in distinguishing between stationary and moving objects and can be used by ADAS systems for emergency braking assistance and forward collision warning. Long-range RADAR sensors can include monostatic multimodal RADAR with multiple (e.g., six or more) fixed RADAR antennas and a high-speed CAN and FlexRay interface.In an example with six antennas, the four middle antennas can generate a focused beam pattern used to record the vehicle's surroundings at higher speeds with minimal interference from traffic in adjacent lanes. The other two antennas can widen the field of view, allowing vehicles entering or leaving the vehicle's lane to be detected quickly.

[0161] Medium-range radar systems, for example, can have a range of up to 1060 m (front) or 80 m (rear) and a field of view of up to 42 degrees (front) or 1050 degrees (rear). Short-range radar systems can include, among other things, radar sensors designed for installation at both ends of the rear bumper. When installed at both ends of the rear bumper, such radar sensor systems can generate two beams that continuously monitor the blind spots behind and beside the vehicle.

[0162] Short-range radar systems can be used in an ADAS system for blind spot detection and / or lane change assistance.

[0163] The vehicle 1000 can also include ultrasonic sensors 1062. The ultrasonic sensors 1062, which can be positioned on the front, rear, and / or sides of the vehicle 1000, can be used for parking assistance and / or for creating and updating an occupancy grid. A variety of ultrasonic sensors 1062 can be used, and different ultrasonic sensors 1062 can be used for different detection ranges (e.g., 2.5 m, 4 m). The ultrasonic sensors 1062 can be operated with a functional safety level of ASIL B.

[0164] The vehicle 1000 can include one or more LiDAR sensors 1064. The LiDAR sensor(s) 1064 can be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LiDAR sensor(s) 1064 can have functional safety level ASIL B. In some examples, the vehicle 1000 can include multiple LiDAR sensors 1064 (e.g., two, four, six, etc.) that can use Ethernet (e.g., to deliver data to a Gigabit Ethernet switch).

[0165] In some examples, the LiDAR sensors 1064 may be able to provide a list of objects and their distances for a 360-degree field of view. Commercially available LiDAR sensors 1064 may, for example, have a specified range of approximately 1000 m with an accuracy of 2 cm to 3 cm and support for a 1000 Mbps Ethernet connection. In some examples, one or more non-protruding LiDAR sensors 1064 may be used. In such examples, the LiDAR sensors 1064 may be implemented as a small device that can be embedded in the front, rear, sides, and / or corners of the vehicle 1000. The LIDAR sensor(s) 1064 can provide a horizontal field of view of up to 120 degrees and a vertical field of view of up to 35 degrees with a range of 200 m in such examples, even for objects with low reflectivity.The front-mounted LIDAR sensor(s) 1064 can be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0166] In some examples, LiDAR technologies such as 3D flash LiDAR can also be used. 3D flash LiDAR uses a laser pulse as a transmission source to illuminate the vehicle's surroundings up to a distance of approximately 200 m. A flash LiDAR unit includes a sensor that records the travel time of the laser pulse and the reflected light at each pixel, which in turn corresponds to the distance from the vehicle to the objects. Flash LiDAR can generate highly accurate and distortion-free images of the surroundings with each laser pulse. In some examples, four flash LiDAR sensors can be used, one on each side of the vehicle. Available 3D flash LiDAR systems include a solid-state 3D staring array LiDAR camera with no moving parts except for a fan (e.g., a non-scanning device).The flash LIDAR device can use a 5-nanosecond Class I (eye-safe) laser pulse per image and capture the reflected laser light in the form of 3D distance point clouds and co-recorded intensity data. Due to the use of flash LIDAR, and because flash LIDAR is a solid-state device with no moving parts, the LIDAR 1064 sensors may be less susceptible to motion blur, vibration, and / or shock.

[0167] The vehicle may further include IMU sensors 1066. In some examples, the IMU sensors 1066 may be located in the center of the rear axle of the vehicle 1000. The IMU sensor(s) 1066 may, for example, and without limitation, include one or more accelerometers, one or more magnetometers, one or more gyroscopes, one or more magnetic compasses, and / or other sensor types. In some examples, such as in six-axis applications, the IMU sensors 1066 may include accelerometers and gyroscopes, while in nine-axis applications, the IMU sensors 1066 may include accelerometers, gyroscopes, and magnetometers.

[0168] In some embodiments, the IMU sensor(s) 1066 can be implemented as a miniaturized, high-performance GPS-based inertial navigation system (GPS / INS) that combines microelectromechanical systems (MEMS) inertial sensors, a high-sensitivity GPS receiver, and advanced Kalman filter algorithms to provide estimates of position, velocity, and orientation. In some examples, the IMU sensor(s) 1066 can enable the vehicle 1000 to estimate its direction of travel without requiring input from a magnetic sensor by directly observing velocity changes from the GPS and correlating them with the IMU sensor(s) 1066. In some examples, the IMU sensor(s) 1066 and the GNSS sensor(s) 1058 can be combined in a single integrated unit.

[0169] The vehicle can include one or more microphones 1096, which are arranged in and / or around the vehicle 1000. The microphones 1096 can be used, among other things, for the detection and identification of emergency vehicles.

[0170] The vehicle may further include any number of camera types, including stereo camera(s) 1068, wide-angle camera(s) 1070, infrared camera(s) 1072, surround camera(s) 1074, long-range and / or medium-range camera(s) 1098, and / or other camera types. The cameras may be used to capture image data around the entire perimeter of the vehicle 1000. The camera types used depend on the embodiment and requirements for the vehicle 1000, and any combination of camera types may be used to provide the required coverage around the vehicle 1000. Furthermore, the number of cameras may vary depending on the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. The cameras may, for example, and without limitation, support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet.Each of the cameras is described herein in relation to . Fig. 10A and Fig. 10B described in more detail.

[0171] The vehicle 1000 may further include one or more vibration sensors 1042. The vibration sensor(s) 1042 can measure vibrations of vehicle components, such as the axle(s). For example, changes in vibrations may indicate a change in the road surface. In another example, when using two or more vibration sensors 1042, the differences between the vibrations can be used to determine the friction or slippage of the road surface (e.g., when the difference in vibration is between a driven axle and a freely rotating axle).

[0172] Vehicle 1000 may include an ADAS system 1038. In some examples, the ADAS system 1038 may include a system of computation (SoC). The ADAS system 1038 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keeping assist (LKA), blind spot warning (BSW), rear cross traffic alert (RCTW), collision warning systems (CWS), lane centering (LC), and / or other features and functions.

[0173] The ACC systems can use radar sensors (1060), lidar sensors (1064), and / or cameras. The ACC systems can include longitudinal ACC and / or lateral ACC. Longitudinal ACC monitors and controls the distance to the vehicle immediately in front and automatically adjusts the vehicle speed to maintain a safe distance from vehicles ahead. Lateral ACC ensures that the distance is maintained and, if necessary, instructs the vehicle to change lanes. Lateral ACC is related to other ADAS applications such as LCA and CWS.

[0174] CACC uses information from other vehicles, which can be received via the network interface 1024 and / or the wireless antennas 1026, either wirelessly or indirectly via a network connection (e.g., the internet). Direct connections can be established through a vehicle-to-vehicle (V2V) communication link, while indirect connections can be an infrastructure-to-vehicle (I2V) communication link. Generally, the V2V communication concept provides information about the vehicles immediately ahead (e.g., vehicles directly in front of vehicle 1000 and in the same lane), while the I2V communication concept provides information about traffic further ahead. CACC systems can incorporate either I2V or V2V information sources, or both.Given the information about the 1000 vehicles driving in front of the vehicle, CACC can be more reliable and has the potential to improve traffic flow and reduce congestion on the road.

[0175] FCW systems are designed to alert the driver to a hazard so they can take corrective action. FCW systems use a forward-facing camera and / or RADAR 1060 sensors coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically connected to driver feedback such as a display, speaker, and / or vibrating component. FCW systems can issue a warning, for example, in the form of an audible or visual signal, a vibration, and / or a brief braking pulse.

[0176] AEB systems detect an impending frontal collision with another vehicle or object and can automatically apply the brakes if the driver does not take corrective action within a specified time or distance parameter. AEB systems may use forward-facing camera(s) and / or radar sensor(s) coupled with a dedicated processor, DSP, FPGA, and / or ASIC. When the AEB system detects a hazard, it typically first warns the driver so they can take corrective action to avoid the collision. If the driver does not take corrective action, the AEB system can automatically apply the brakes to prevent or at least mitigate the effects of the predicted collision. AEB systems may include techniques such as dynamic brake assist and / or emergency braking in the event of an impending impact.

[0177] Lane Departure Warning (LDW) systems provide visual, audible, and / or tactile warnings, such as vibrations in the steering wheel or seat, to alert the driver if the vehicle crosses lane markings. An LDW system will not activate if the driver signals an intentional lane departure by using the turn signal. LDW systems may use forward-facing cameras coupled with a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically connected to driver feedback, such as a display, speaker, and / or vibrating component.

[0178] Lane Keeping Assist (LKA) systems are a type of Lane Departure Warning (LDW) system. LKA systems correct the steering or brake the vehicle if it begins to leave its lane.

[0179] Blind Spot Warning (BSW) systems detect vehicles in a car's blind spot and warn the driver. BSW systems can issue visual, audible, and / or tactile warnings to indicate that merging or changing lanes is unsafe. The system can issue an additional warning if the driver activates the turn signal. BSW systems can use rear-facing camera(s) and / or radar sensor(s) coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically connected to driver feedback, such as a display, speaker, and / or vibrating component.

[0180] RCTW systems can issue visual, audible, and / or tactile warnings when an object outside the reversing camera's field of view is detected while the vehicle is reversing. Some RCTW systems include AEB to ensure the vehicle's brakes are applied to avoid a collision. RCTW systems can use one or more rear-facing radar sensors coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically connected to provide feedback to the driver, such as a display, speaker, and / or vibrating component.

[0181] Conventional ADAS systems can be prone to false alarms, which can be disruptive and distracting for the driver, but are generally not catastrophic because the ADAS systems warn the driver and leave it to them to decide whether a safety condition actually exists and to act accordingly. In an autonomous vehicle 1000, however, the vehicle 1000 itself must decide, in the case of conflicting results, whether to heed the result of a primary computer or a secondary computer (e.g., a first controller 1036 or a second controller 1036). In some embodiments, the ADAS system 1038, for example, can be a backup and / or secondary computer that provides perceptual information to a backup computer rationality module. The backup computer rationality monitor can run redundant, diverse software on hardware components to detect errors in perception and dynamic driving tasks.The outputs of the ADAS system 1038 can be forwarded to a higher-level MCU. If the outputs of the primary and secondary computers conflict, the higher-level MCU must decide how to resolve the conflict to ensure safe operation.

[0182] In some examples, the primary computer may be configured to provide the higher-level MCU with a confidence score indicating its confidence in the chosen result. If the confidence score exceeds a threshold, the higher-level MCU can follow the primary computer's instruction, regardless of whether the secondary computer returns a conflicting or inconsistent result. If the confidence score does not reach the threshold, and if the primary and secondary computers display different results (e.g., conflicting results), the higher-level MCU can mediate between the computers to determine the appropriate result.

[0183] The monitoring MCU can be configured to run a neural network trained and configured to determine, based on the outputs of the primary and secondary computers, the conditions under which the secondary computer generates false alarms. Thus, the neural network in the monitoring MCU can learn when the secondary computer's output is trustworthy and when it is not. For example, if the secondary computer is a radar-based FCW system, a neural network in the parent MCU can learn when the FCW system identifies metallic objects that do not actually pose a hazard, such as a drain grate or manhole cover, triggering an alarm.If the secondary computer is a camera-based lane departure warning (LDW) system, a neural network in the higher-level MCU can also learn to override the LDW when cyclists or pedestrians are present and a lane change is indeed the safest maneuver. In embodiments comprising one or more neural networks running on the higher-level MCU, the higher-level MCU may include at least one DLA or GPU suitable for operating the neural network(s) with associated memory. In preferred embodiments, the higher-level MCU may be included as a component of and / or contained within the SoC(s) 1004.

[0184] In other examples, the ADAS system 1038 may include a secondary computer that performs ADAS functions using traditional computer vision rules. As such, the secondary computer may employ classic computer vision rules (if-then), and the presence of one or more neural networks in the parent MCU may enhance reliability, safety, and performance. For instance, the diverse implementation and intentional non-identity make the overall system more fault-tolerant, particularly to errors caused by software functions (or software-hardware interfaces).For example, if there is a software bug or a fault in the software running on the primary computer, and the non-identical software code running on the secondary computer produces the same overall result, the higher-level MCU can be more confident that the overall result is correct and that the fault in the software or hardware on the primary computer is not causing a material error.

[0185] In some cases, the output of the ADAS system 1038 can be fed into the perception block and / or the dynamic driving task block of the main computer. For example, if the ADAS system 1038 issues a forward collision warning due to an object located directly in front of the vehicle, the perception block can use this information in object identification. In other examples, the secondary computer may have its own trained neural network, thus reducing the risk of false alarms, as described herein.

[0186] The Vehicle 1000 may further include the Infotainment SoC 1030 (e.g., an in-vehicle infotainment system (IVI)). Although depicted and described as an SoC, the infotainment system need not be an SoC and may comprise two or more discrete components. The Infotainment SoC 1030 may include a combination of hardware and software that can be provided to the Vehicle 1000 with audio (e.g., music, a personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., television, movies, streaming, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and / or information services (e.g., navigation systems, rear parking assistance, a radio data system, vehicle-related information such as fuel level, total distance traveled, brake fluid level, oil level, door open / close status, air filter information, etc.).For example, the Infotainment SoC 1030 can include radios, disc players, navigation systems, video players, USB and Bluetooth connectivity, car computers, in-vehicle entertainment systems, Wi-Fi, steering wheel audio controls, hands-free voice control, a head-up display (HUD), an HMI display 1034, a telematics device, a control panel (e.g., for controlling and / or interacting with various components, functions, and / or systems), and / or other components. The Infotainment SoC 1030 can also be used to provide information (e.g., visual and / or audible) to one or more vehicle users, such as information from the ADAS system 1038, autonomous driving information like planned vehicle maneuvers, trajectories, environmental information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0187] The infotainment SoC 1030 may include GPU functionality. The infotainment SoC 1030 can communicate with other devices, systems, and / or components of the vehicle 1000 via the bus 1002 (e.g., CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 1030 may be coupled with a higher-level MCU so that the infotainment system's GPU can perform some self-driving functions if the primary controller(s) 1036 (e.g., the primary and / or backup computer of the vehicle 1000) fails. In such an example, the infotainment SoC 1030 can put the vehicle 1000 into a driver-to-safety stop mode, as described herein.

[0188] The vehicle 1000 may further include an instrument cluster 1032 (e.g., a digital instrument panel, an electronic instrument cluster, a digital instrument panel, etc.). The instrument cluster 1032 may include a controller and / or a supercomputer (e.g., a discrete controller or supercomputer). The instrument cluster 1032 may include a number of gauges, such as a speedometer, fuel gauge, oil pressure gauge, tachometer, odometer, turn signals, gear position indicator, seat belt warning lights, handbrake warning lights, engine malfunction lights, airbag system information (SRS), lighting controls, safety system controls, navigation information, etc. In some examples, information may be displayed and / or exchanged between the infotainment SoC 1030 and the instrument cluster 1032. In other words, the instrument cluster 1032 may be included as part of the infotainment SoC 1030, or vice versa.

[0189] Fig. 10D is a system diagram for communication between cloud-based servers and the autonomous vehicle 1000 from Fig. 10A according to some embodiments of the present disclosure. The system 1076 may comprise servers 1078, networks 1090, and vehicles, including the vehicle 1000. The servers 1078 may comprise a variety of GPUs 1084(A)-1084(H) (hereinafter collectively referred to as GPUs 1084), PCIe switches 1082(A)-1082(H) (hereinafter collectively referred to as PCIe switches 1082), and / or CPUs 1080(A)-1080(B) (hereinafter collectively referred to as CPUs 1080). The GPUs 1084, the CPUs 1080, and the PCIe switches can be interconnected via high-speed links, such as, but not limited to, NVIDIA's NVLink 1088 interfaces and / or PCIe 1086 links. In some examples, the GPUs 1084 are connected via NVLink and / or NVSwitch SoC, and the GPUs 1084 and the PCIe 1082 switches are connected via PCIe links.Although eight GPUs 1084, two CPUs 1080, and two PCIe switches are shown, this is not intended to be a limitation. Depending on the configuration, each Server 1078 can include any number of GPUs 1084, CPUs 1080, and / or PCIe switches. For example, each Server 1078 can include eight, sixteen, thirty-two, and / or more GPUs 1084.

[0190] The server(s) 1078 can receive image data from the network(s) 1090 and the vehicles, depicting unexpected or altered road conditions, such as recently started roadworks. The server(s) 1078 can transmit neural networks 1092, updated neural networks 1092, and / or map information 1094, including traffic and road condition information, to the vehicles via the network(s) 1090 and the vehicles. Map information updates 1094 may include updates to the HD map 1022, such as information about construction sites, potholes, detours, flooding, and / or other obstacles.In some examples, the neural networks 1092, the updated neural networks 1092 and / or the map information 1094 may result from new training and / or experience represented in data received from any number of vehicles in the environment, and / or based on training performed in a data center (e.g. using server(s) 1078 and / or other servers).

[0191] Server 1078 can be used to train machine learning models (e.g., neural networks) based on training data. The training data can be generated by the vehicles and / or in a simulation (e.g., using a game engine). In some examples, the training data is tagged (e.g., if the neural network benefits from supervised learning) and / or subjected to other preprocessing, while in other examples, the training data is not tagged and / or preprocessed (e.g., if the neural network does not require supervised learning).Training can be performed according to one or more classes of machine learning techniques, including but not limited to: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, federated learning, transfer learning, feature learning (including principal component and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including spare dictionary learning), rule-based machine learning, anomaly detection, and all variants or combinations thereof. Once the machine learning models are trained, they can be used by the vehicles (e.g., transmitted to the vehicles via network(s) 1090) and / or they can be used by the server(s) 1078 for remote monitoring of the vehicles.

[0192] In some examples, the Server 1078 can receive data from the vehicles and apply the data to current real-time neural networks for intelligent real-time inference. The Server 1078 can include deep learning supercomputers and / or dedicated AI computers powered by GPU(s) 1084, such as the DGX and DGX Station machines developed by NVIDIA. However, in some examples, the Server 1078 can include a deep learning infrastructure that uses only CPU-powered data centers.

[0193] The deep learning infrastructure of server(s) 1078 may be able to perform fast, real-time inference and use this capability to assess and verify the health of the processors, software, and / or associated hardware in vehicle 1000. For example, the deep learning infrastructure may receive periodic updates from vehicle 1000, such as a sequence of images and / or objects that vehicle 1000 has located within that sequence (e.g., using computer vision and / or other machine learning methods for object classification). The deep learning infrastructure may then operate its own neural network to identify these objects and compare them to those identified by vehicle 1000.If the results do not match and the infrastructure indicates that the AI ​​in vehicle 1000 is malfunctioning, the servers 1078 can send a signal to vehicle 1000, instructing a fail-safe computer in vehicle 1000 to take control, notify the occupants, and perform a safe parking maneuver.

[0194] For inference, the server(s) 1078 can include the GPU(s) 1084 and one or more programmable inference accelerators (e.g., NVIDIA's TensorRT). The combination of GPU-powered servers and inference acceleration can enable real-time responsiveness. In other scenarios, such as when performance is less critical, servers with CPUs, FPGAs, and other processors can be used for inference. EXAMPLE OF A COMPUTER DEVICE

[0195] Fig. Figure 11 is a block diagram of an example of one or more devices 1100 suitable for implementing some embodiments of the present disclosure. The device 1100 may comprise a connection system 1102 that directly or indirectly connects the following devices: a memory 1104, one or more central processing units (CPUs) 1106, one or more graphics processing units (GPUs) 1108, a communication interface 1110, input / output (I / O) ports 1112, input / output components 1114, a power supply 1116, one or more presentation components 1118 (e.g., display(s)), and one or more logic units 1120. In at least one embodiment, the device 1100 may comprise one or more virtual machines (VMs), and / or each of its components may comprise virtual components (e.g., virtual hardware components).As non-restrictive examples, one or more of the GPUs 1108 can comprise one or more vGPUs, one or more of the CPUs 1106 can comprise one or more vCPUs, and / or one or more of the logic units 1120 can comprise one or more virtual logic units. Thus, one or more devices 1100 can comprise discrete components (e.g., a complete GPU allocated to the device 1100), virtual components (e.g., a portion of a GPU allocated to the device 1100), or a combination thereof.

[0196] Although the various blocks in Fig. The fact that components 11 are shown as being connected via the connection system 1102 is not intended to be a limitation and serves only for clarity. For example, in some embodiments, a presentation component 1118, such as a display device, can be considered an I / O component 1114 (e.g., if the display is a touchscreen). As another example, the CPUs 1106 and / or GPUs 1108 can contain memory (e.g., the memory 1104 can represent a storage device in addition to the memory of the GPUs 1108, the CPUs 1106, and / or other components). In other words, the computing device of Fig. Section 11 is for illustrative purposes only. No distinction is made between categories such as "workstation", "server", "laptop", "desktop", "tablet", "client device", "mobile device", "handheld device", "game console", "electronic control unit (ECU)", "virtual reality system" and / or other device or system types, as all fall under the category of computing device. Fig. 11 are being considered.

[0197] The connection system 1102 can represent one or more connections or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The connection system 1102 can include one or more bus or connection types, such as an ISA (Industry Standard Architecture) bus, an EISA (Extended Industry Standard Architecture) bus, a VESA (Video Electronics Standards Association) bus, a PCI (Peripheral Component Interconnect) bus, a PCIe (Peripheral Component Interconnect Express) bus, and / or another bus or connection type. In some embodiments, there are direct connections between components. For example, the CPU 1106 can be directly connected to the memory 1104. Furthermore, the CPU 1106 can be directly connected to the GPU 1108.If a direct or point-to-point connection exists between components, the 1102 connection system can include a PCIe connection to establish the connection. In these examples, a PCI bus does not need to be included in the 1100 computing unit.

[0198] The memory 1104 can comprise a variety of computer-readable media. The computer-readable media can be any available media that the device 1100 can access. The computer-readable media can include both volatile and non-volatile media, as well as removable and non-removable media. For example, and without limitation, the computer-readable media can include computer storage media and communication media.

[0199] Computer storage media can include both volatile and non-volatile media, and / or removable and non-removable media, implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 1104 can store computer-readable instructions (e.g., those representing a program and / or a program element, such as an operating system). Computer storage media can include, but are not limited to, RAM, ROM, EEPROM, flash memory or other storage technologies, CD-ROM, digital versatile discs (DVDs) or other optical storage media, magnetic cartridges, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that the computer device 1100 can access.As used here, computer storage media do not contain signals per se.

[0200] Computer storage media can contain computer-readable instructions, data structures, program modules, and / or other data types in a modulated data signal, such as a carrier wave or other transport mechanism, and encompasses all information transmission media. The term "modulated data signal" can refer to a signal in which one or more properties have been set or modified so that information is encoded in the signal. For example, and without limitation, computer storage media can include wired media, such as a wired network or a directly wired connection, and wireless media, such as acoustic, RF, infrared, and other wireless media. Combinations of the aforementioned media should also fall under the category of computer-readable media.

[0201] The CPU(s) 1106 can be configured to execute at least some of the computer-readable instructions to control one or more components of the device 1100 so that it executes one or more of the procedures and / or processes described herein. The CPU(s) 1106 can each comprise one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of processing a plurality of software threads simultaneously. The CPU(s) 1106 can comprise any processor type and may include different processor types depending on the type of computer device 1100 being implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers).Depending on the type of computing device 1100, the processor can be, for example, an Advanced RISC Machines (ARM) processor implemented using Reduced Instruction Set Computing (RISC), or an x86 processor implemented using Complex Instruction Set Computing (CISC). The device 1100 can include one or more CPUs 1106 in addition to one or more microprocessors or supplementary coprocessors, such as mathematical coprocessors.

[0202] In addition to or as an alternative to the CPU(s) 1106, the GPU(s) 1108 may be configured to execute at least some of the computer-readable instructions to control one or more components of the device 1100 so that they perform one or more of the methods and / or processes described herein. One or more of the GPU(s) 1108 may be an integrated GPU (e.g., with one or more of the CPU(s) 1106 and / or one or more of the GPU(s) 1108) or a discrete GPU. In embodiments, one or more of the GPU(s) 1108 may be a coprocessor of one or more of the CPU(s) 1106. The GPU(s) 1108 may be used by the device 1100 for rendering graphics (e.g., 3D graphics) or for performing general-purpose calculations. For example, the GPU(s) 1108 may be used for general-purpose GPU computing (GPGPU).The GPU(s) 1108 can comprise hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. The GPU(s) 1108 can generate pixel data for output images in response to rendering commands (e.g., rendering commands from the CPU(s) 1106 received via a host interface). The GPU(s) 1108 can include graphics memory, such as display memory, for storing pixel data or other suitable data, such as GPGPU data. The display memory can be included as part of the memory 1104. The GPU(s) 1108 can comprise two or more GPUs operating in parallel (e.g., via a link). The link can connect the GPUs directly (e.g., using NVLINK) or connect them via a switch (e.g., using NVSwitch).When combined, each GPU can generate 1108 pixel data or GPGPU data for different parts of an output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU can have its own dedicated memory or share memory with other GPUs.

[0203] In addition to or as an alternative to the CPU(s) 1106 and / or the GPU(s) 1108, the logic unit(s) 1120 may be configured to execute at least some of the computer-readable instructions to control one or more components of the device 1100 to execute one or more of the methods and / or processes described herein. In embodiments, the CPU(s) 1106, the GPU(s) 1108, and / or the logic unit(s) 1120 may execute any combination of the methods, processes, and / or parts thereof discretely or jointly. One or more of the logic units 1120 may be part of and / or integrated into one or more of the CPUs 1106 and / or GPUs 1108, and one or more of the logic units 1120 may be discrete components or otherwise separate from the CPUs 1106 and / or GPUs 1108.In embodiments, one or more of the logic units 1120 can be a coprocessor of one or more of the CPUs 1106 and / or one or more of the GPUs 1108.

[0204] Examples of the logic unit(s) 1120 include one or more processor cores and / or components thereof, such as data processing units (DPUs), tensor cores (TCs), tensor processing units (TPUs), pixel visual cores (PVCs), image processing units (VPUs), graphics processing clusters (GPCs), texture processing clusters (TPCs), streaming multiprocessors (SMs), tree traversing units (TTUs), artificial intelligence accelerators (AIAs), deep learning accelerators (DLAs), arithmetic logic units (ALUs), application-specific integrated circuits (ASICs), floating-point units (FPUs), input / output (I / O) elements, peripheral component interconnect (PCI) or peripheral component interconnect express (PCIe) elements, and / or the like.

[0205] The communication interface 1110 can include one or more receivers, transmitters, and / or transceivers that enable the device 1100 to communicate with other computer devices via an electronic communication network, including wired and / or wireless communication. The communication interface 1110 can include components and functions that enable communication over a variety of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication via Ethernet or InfiniBand), low-power wide-area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet.In one or more embodiments, the logic unit(s) 1120 and / or the communication interface 1110 may include one or more data processing units (DPUs) to directly transfer data received via a network and / or the connection system 1102 to one or more GPUs 1108 (e.g., a memory thereof).

[0206] The I / O ports 1112 enable the device 1100 to be logically coupled with other devices, including the I / O components 1114, the presentation component(s) 1118, and / or other components, some of which may be built into (e.g., integrated with) the device 1100. Examples of I / O components 1114 include a microphone, mouse, keyboard, joystick, gamepad, game controller, satellite dish, scanner, printer, wireless device, etc. The I / O components 1114 can provide a natural user interface (NUI) that processes air gestures, speech, or other physiological input from a user. In some cases, input can be transferred to a suitable network element for further processing.A NUI can implement any combination of speech recognition, pen recognition, facial recognition, biometric recognition, gesture recognition (both on-screen and beside the screen), air gestures, head and eye tracking, and touch recognition (as described in more detail below) in conjunction with a display of the device 1100. The device 1100 can include depth cameras, such as stereoscopic camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof, for gesture detection and recognition. Additionally, the device 1100 can include accelerometers or gyroscopes (e.g., as part of an inertial measurement unit (IMU)) that enable motion detection. In some examples, the output from the accelerometers or gyroscopes can be used by the device 1100 to display immersive augmented reality or virtual reality.

[0207] The power supply 1116 can be a hardwired power supply, a battery power supply, or a combination of both. The power supply 1116 can supply power to the computer device 1100 so that the components of the computer device 1100 can operate.

[0208] The presentation component(s) 1118 may include a display (e.g., a monitor, touchscreen, television screen, head-up display (HUD), other display types, or a combination thereof), speakers, and / or other presentation components. The presentation component(s) 1118 may receive data from other components (e.g., the GPU(s) 1108, the CPU(s) 1106, DPUs, etc.) and output the data (e.g., as an image, video, sound, etc.). EXAMPLE OF A DATA CENTER

[0209] Fig. Figure 12 shows an example of a data center 1200 that can be used in at least one embodiment of the present disclosure. The data center 1200 can comprise a data center infrastructure layer 1210, a framework layer 1220, a software layer 1230, and / or an application layer 1240.

[0210] As in Fig. As shown in Figure 12, the data center infrastructure layer 1210 can include a resource orchestrator 1212, clustered compute resources 1214 and node compute resources (“node CRs”) 1216(1)-1216(N), where “N” is any positive integer. In at least one embodiment, Node CRs 1216(1)-1216(N) can include, among other things, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field-programmable gate arrays (FPGAs), graphics processing units or graphics processing units (GPUs), etc.), storage devices (e.g., dynamic read-only storage), storage devices (e.g., solid-state or hard disk drives), network input / output devices (NW I / O devices), network switches, virtual machines (VMs), power modules and / or cooling modules, etc. In some embodiments, one or more Node CRs can be composed of the Node CRss 1216(1)-1216(N) correspond to a server that has one or more of the aforementioned computing resources. Furthermore, in some embodiments, the node CRs 1216(1)-12161(N) may comprise one or more virtual components, such as vGPUs, vCPUs, and / or the like, and / or one or more of the node CRs 1216(1)-1216(N) may correspond to a virtual machine (VM).

[0211] In at least one embodiment, grouped compute resources 1214 can comprise separate groupings of node CRs 1216 located in one or more racks (not shown) or in many racks located in data centers at different geographic locations (also not shown). Separate groupings of node CRs 1216 within the grouped compute resources 1214 can comprise grouped compute, network, storage, or memory resources that can be configured or allocated to support one or more workloads. In at least one embodiment, multiple node CRs 1216, including CPUs, GPUs, DPUs, and / or other processors, can be grouped within one or more racks to provide grouped compute resources to support one or more workloads.The one or more racks can also include any number of power supply modules, cooling modules and / or network switches in any combination.

[0212] The resource orchestrator 1212 can configure or otherwise control one or more node CRs 1216(1)-1216(N) and / or grouped computing resources 1214. In at least one embodiment, the resource orchestrator 1212 can include a software design infrastructure (SDI) management unit for the data center 1200. The resource orchestrator 1212 can comprise hardware, software, or a combination thereof.

[0213] In at least one embodiment, as in Fig. As shown in Figure 12, the framework layer 1220 can include a job scheduler 1233, a configuration manager 1234, a resource manager 1236, and / or a distributed file system 1238. The framework layer 1220 can include a framework to support the software 1232 of the software layer 1230 and / or one or more applications 1242 of the application layer 1240. The software 1232 or the applications 1242 can each include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. The framework layer 1220 can be, but is not limited to, a type of free and open-source software web application framework such as Apache Spark™ (hereinafter "Spark"), which can utilize a distributed file system 1238 for processing large amounts of data (e.g., "Big Data").In at least one embodiment, the job scheduler 1233 can include a Spark driver to facilitate the scheduling of workloads supported by different layers of the data center 1200. The configuration manager 1234 can be able to configure different layers, such as the software layer 1230 and the framework layer 1220, including Spark and the distributed file system 1238, to support the processing of large amounts of data. The resource manager 1236 can be able to manage clustered or grouped compute resources that are associated with or allocated to support the distributed file system 1238 and the job scheduler 1233. In at least one embodiment, clustered or grouped compute resources can include grouped compute resources 1214 on the data center infrastructure layer 1210.The resource manager 1236 can coordinate with the resource orchestrator 1212 to manage these allocated or assigned computing resources.

[0214] In at least one embodiment, the software 1230 contained in software layer 1232 may comprise software used by at least parts of the node CRs 1216(1)-1216(N), grouped computing resources 1214, and / or the distributed file system 1238 of framework layer 1220. One or more types of software may include, among others, internet website search software, email virus scanning software, database software, and streaming video content software.

[0215] In at least one embodiment, the applications 1242 contained in the application layer 1240 can comprise one or more types of applications used by at least parts of the node CRs 1216(1)-1216(N), grouped compute resources 1214, and / or the distributed file system 1238 of the framework layer 1220. One or more types of applications can include, among others, any number of genomics applications, cognitive computing applications, and machine learning applications, including training or inference software, machine learning software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments.

[0216] In at least one embodiment, each of the configuration manager 1234, resource manager 1236, and resource orchestrator 1212 can implement any number and type of self-modifying actions based on any set and type of data acquired in a technically feasible manner. Self-modifying actions can relieve a data center operator 1200 of potentially making poor configuration decisions and potentially prevent underutilization and / or poor performance of parts of a data center.

[0217] The Data Center 1200 may include tools, services, software, or other resources to train one or more machine learning models or to predict or derive information using one or more machine learning models according to one or more embodiments described herein. For example, a machine learning model may be trained by calculating weight parameters according to a neural network architecture using software and / or computing resources described above in relation to the Data Center 1200.In at least one embodiment, trained or deployed machine learning models corresponding to one or more neural networks can be used to derive or predict information using the resources described above in relation to the data center 1200, by using weight parameters calculated by one or more training techniques, such as, but not limited to, those described herein.

[0218] In at least one embodiment, the data center can use 1200 CPUs, application-specific integrated circuits (ASICs), GPUs, FPGAs, and / or other hardware (or equivalent virtual computing resources) to perform training and / or inference using the resources described above. Furthermore, one or more of the software and / or hardware resources described above can be configured as a service to allow users to train or infer information, such as image recognition, speech recognition, or other artificial intelligence services. EXEMPLARY NETWORK ENVIRONMENTS

[0219] Network environments suitable for implementing embodiments of the disclosure may include one or more client devices, servers, network attached storage (NAS), other backend devices, and / or other device types. The client devices, servers, and / or other devices (e.g., each device) may be located on one or more instances of the computing devices 1100. Fig. 11 be implemented - e.g., each device may include similar components, features, and / or functions of the computing devices 1100. If backend devices (e.g., servers, NAS, etc.) are implemented, the backend devices may also be included as part of a data center 1200, an example of which is given herein in relation to Fig. 12 is described in more detail.

[0220] Components in a network environment can communicate with each other over one or more networks, which can be wired, wireless, or both. The network can comprise multiple networks or a network of networks. For example, the network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks such as the internet and / or a public switched telephone network (PSTN), and / or one or more private networks. If the network includes a wireless telecommunications network, components such as a base station, a communications tower, or even access points (as well as other components) can provide wireless connectivity.

[0221] Compatible network environments can include one or more peer-to-peer network environments—in which case no server may be included in a network environment—and one or more client-server network environments—in which case one or more servers may be included in a network environment. In peer-to-peer network environments, the functions described herein with respect to one or more servers can be implemented on any number of client devices.

[0222] In at least one embodiment, a network environment can comprise one or more cloud-based network environments, a distributed computing environment, a combination thereof, etc. A cloud-based network environment can comprise a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. A framework layer can comprise a framework for supporting software of a software layer and / or one or more applications of an application layer. The software or applications may each include web-based service software or applications. In embodiments, one or more of the client devices can use the web-based service software or applications (e.g.,by accessing the service software and / or applications via one or more application programming interfaces (APIs). The framework layer can be, but is not limited to, a type of free and open-source software web application framework, such as one that can use a distributed file system for processing large amounts of data (e.g., "Big Data").

[0223] A cloud-based network environment can provide cloud computing and / or cloud storage, performing any combination (or one or more parts) of the computing and / or data storage functions described here. Each of these different functions can be distributed across multiple locations by central or core servers (e.g., one or more data centers, which may be distributed across a state, region, country, the globe, etc.). If a connection to a user (e.g., a client device) is relatively close to one or more edge servers, one or more core servers can delegate at least some of the functionality to the edge server(s). A cloud-based network environment can be private (e.g., restricted to a single organization), public (e.g., available to many organizations), and / or a combination of both (e.g., a hybrid cloud environment).

[0224] The client device(s) can / can include at least some of the components, features, and functions of the information referred to here. Fig.The 11 described example computing devices include 1100. For example, and without limitation, a client device may be a personal computer (PC), laptop, mobile device, smartphone, tablet computer, smartwatch, portable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, video camera, surveillance device or system, vehicle, boat, aircraft, virtual machine, drone, robot, handheld communication device, hospital device, gaming device or system, entertainment system, vehicle computer system, embedded system controller, remote control, household appliance, consumer electronics device, workstation, edge device, any combination of these described devices, or any other suitable device.

[0225] Disclosure can be described in the general context of computer code or machine-readable instructions, including computer-executable instructions such as program modules, which are executed by a computer or other machine, such as a personal digital assistant or other handheld device. In general, program modules, including routines, programs, objects, components, data structures, etc., refer to code that performs specific tasks or implements certain abstract data types. Disclosure can be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing equipment, etc. Disclosure can also be practiced in distributed computing environments, where distributed tasks are performed by remote processing devices interconnected via a communication network.

[0226] As used here, a statement of "and / or" in relation to two or more elements should be interpreted as meaning only one element or a combination of elements. For example, "element A, element B and / or element C" can include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. Furthermore, "at least one of element A or element B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Additionally, "at least one of element A and B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0227] The subject matter of this disclosure is specifically described herein to comply with legal requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors have considered that the claimed subject matter could also be embodied in other ways to include various steps or combinations of steps similar to those described in this document, in conjunction with other present or future technologies. Although the terms "step" and / or "block" may be used here to denote various elements of the processes employed, these terms should not be interpreted as implying a specific sequence between the various steps disclosed herein, unless the sequence of each step is expressly described. EXAMPLE PARAGRAPHS A. Method comprising: applying image data representing an image showing one or more traffic light devices as input to one or more deep neural networks (DNNs); determining one or more attributes associated with the one or more traffic light devices using one or more component heads of the one or more DNNs; determining one or more classifications associated with one or more traffic light devices using a fusion head of the one or more DNNs and based on at least one of the one or more attributes; and causing a machine to perform one or more control operations based on at least one of the one or more attributes or the one or more classifications associated with one or more traffic light devices. B. The method according to paragraph A, wherein the one or more features associated with the one or more traffic lighting devices comprise at least one of the following elements: one or more orientations of the one or more traffic lighting devices; one or more housing shapes of the one or more traffic lights; one or more active luminaire colours of the one or more traffic lighting devices; or one or more active luminaire shapes of the one or more traffic lighting devices. C. The method according to any one of paragraphs A to B, wherein the one or more component heads comprise at least one component head that outputs a combination of detected attributes associated with the one or more traffic light devices. D. The method according to one of paragraphs A to C, wherein the attributes of the combination recorded include at least one or more color and shape combinations of one or more active light sources of the one or more traffic lighting devices. E. The method according to any one of paragraphs A to D, wherein the one or more classifications associated with the one or more traffic light devices comprise at least a subset of the one or more attributes, wherein the one or more classifications are determined using the fusion head at least on the basis of a combination of the one or more attributes. F. The method according to any one of paragraphs A to E, further comprising: generating, based on at least one or more DNNs processing the image data, one or more component feature vectors corresponding to the one or more traffic light devices depicted in the image, wherein: the one or more features are determined using the one or more component heads, based at least on the application of the one or more component feature vectors to the one or more component heads, and the one or more features are determined using the fusion head, based at least on the application of a combined feature vector comprising a combination of the one or more component feature vectors to the fusion head. G. The method according to any one of paragraphs A to F, wherein the one or more component heads comprise at least a first component head and a second component head, wherein the first component head classifies one or more first attributes of the one or more traffic lighting devices and the second component head classifies one or more second attributes of the one or more traffic lighting devices. H. A system comprising: one or more processors to: determine, based on at least one or more first layers of a machine learning model processing sensor data obtained using one or more sensors with fields of view or sensor fields comprising a traffic light device, first data corresponding to one or more first attributes associated with the traffic light device; determine, based on at least one or more second layers of the machine learning model processing the first data, second data corresponding to one or more second attributes associated with the traffic light device; and perform one or more machine-associated operations based on at least one of the first attributes or the second attributes. I. The system according to paragraph H, wherein at least one of the first attributes or the one or more second attributes comprise at least one of the following elements: an orientation of the traffic light device; a housing shape of the traffic light device; active luminaire colours of the traffic light device; active luminaire shapes of the traffic light device; a number of luminaires of the traffic light device; a road user of the traffic light device; or a flashing state of the traffic light device. J. The system according to any one of paragraphs H to I, wherein the one or more processors furthermore: determine a classification associated with the traffic light device using one or more fusion layers of the machine learning model, wherein the execution of the one or more operations associated with the machine furthermore is carried out at least on the basis of the classification. K. The system according to one of paragraphs H to J, wherein the determination of the classification associated with the traffic light device is based on a combination of at least one of the following elements: the one or more first attributes and the one or more second attributes; or a first feature vector and one or more second feature vectors, wherein the first feature vector is used as input for the one or more first layers and the one or more second feature vectors are used as input for the one or more second layers. L. The system according to any one of paragraphs H to K, wherein the one or more first attributes determined using the one or more first layers comprise at least one or more color and shape combinations of one or more active light sources of the traffic lighting device. M. The system according to any one of paragraphs H to L, wherein at least one of the one or more first attributes or the one or more second attributes comprises one or more housing shapes connected with the traffic light device, wherein the one or more housing shapes correspond to at least one of the following shapes: a vertical housing shape, a horizontal housing shape, a doghouse housing shape or a pedestrian hybrid beacon housing shape. N. The system according to any one of paragraphs H to M, wherein the machine learning model is trained by at least the following steps: capturing an image representing a second traffic light device; updating one or more parts of the image to produce an updated image representing the second traffic light device with one or more updated attributes; and updating one or more parameters associated with the one or more first layers or the one or more second layers of the machine learning model, based at least on applying the updated image as training input to the machine learning model. O. The system according to any one of paragraphs H to N, wherein the system is comprised of at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing a light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more generative AI operations; a system for performing operations using a large language model;a system for performing operations using one or more Vision Language Models (VLMs); a system for performing operations using one or more multimodal language models; a system for using or providing one or more inference microservices; a system for performing one or more conversational AI operations; a system for generating synthetic data; a system for displaying at least one of the following: virtual reality content, augmented reality content, or mixed reality content; a system comprising one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system that is at least partially implemented using cloud computing resources. P. One or more processors, comprising: a processing circuit for performing one or more operations corresponding to a machine operating on the basis of at least one or more attributes associated with a first traffic light device, wherein the one or more attributes are determined using one or more machine learning models, the one or more machine learning models being trained at least by: receiving an image representing a second traffic light device with one or more first attributes; updating one or more parts of the image to produce an updated image, the updated image representing the second traffic light device with one or more second attributes;and updating one or more parameters associated with one or more component heads of one or more machine learning models, based at least on applying the updated image as training input to the one or more machine learning models. Q. The one or more processors according to paragraph P, wherein updating the one or more parts of the image to produce the updated image comprises modifying one or more values ​​of one or more pixels of the image corresponding to one or more active light sources of the second traffic light device, wherein the one or more values ​​are modified such that the one or more active light sources are represented as inactive light sources in the updated image. R. The one or more processors according to any one of paragraphs P to Q, wherein the image represents the second traffic light device in a first state and the updating of one or more parts of the image to generate the updated image comprises updating the image such that the updated image represents the second traffic light device in a second state which is different from the first state. S. The one or more processors according to any one of paragraphs P to R, wherein the updating of the one or more parts of the image to produce the updated image comprises at least one of the following elements: updating one or more shapes of one or more light sources of the second traffic light device; updating an orientation of the second traffic light device; updating a housing shape of the second traffic light device; or updating a number of light sources connected to the second traffic light device. T. The one or more processors according to any one of paragraphs P to S, wherein the one or more processors are included in at least one of the following systems: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing a light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more generative AI operations; a system for performing operations using a large language model;a system for performing operations using one or more Vision Language Models (VLMs); a system for performing operations using one or more multimodal language models; a system for using or providing one or more inference microservices; a system for performing one or more conversational AI operations; a system for generating synthetic data; a system for displaying at least one of the following: virtual reality content, augmented reality content, or mixed reality content; a system comprising one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system that is at least partially implemented using cloud computing resources.

[0228] It is understood that the aspects and embodiments described above serve only as examples and that detailed modifications may be made within the scope of the claims.

[0229] Each device, method and feature disclosed in the description and (where applicable) in the claims and drawings can be provided independently or in any suitable combination.

[0230] The reference numbers listed in the claims are for illustrative purposes only and do not restrict the scope of the claims. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] US 16 / 101.232

[0130] Cited non-patent literature

[0000] Taxonomy and definitions for terms related to driving systems for motor vehicles on the road” were established. (Standard No. J3016-201806, published on June 15, 2018, Standard No. J3016-201609, published on September 30, 2016

[0088]

Claims

[1] Procedure, encompassing: Applying image data representing an image showing one or more traffic light devices as input for one or more deep neural networks (DNNs); Determining one or more attributes associated with one or more traffic light devices using one or more component heads of one or more DNNs; Determine one or more classifications associated with one or more traffic light devices using a fusion head of one or more DNNs and based on at least one or more attributes; and Causing a machine to perform one or more control operations based on at least one of the one or more attributes or classifications associated with the one or more traffic light devices. [2] Method according to claim 1, wherein the one or more attributes assigned to the one or more traffic light devices comprise at least one of the following elements: one or more orientations of one or more traffic light devices; one or more housing types of one or more traffic lighting devices; one or more active luminous colors of one or more traffic lighting devices; or one or more active light source forms of one or more traffic lighting devices. [3] Method according to claim 1 or 2, wherein the one or more component heads comprise at least one component head configured to output a combination of detected attributes associated with the one or more traffic light devices. [4] Method according to claim 3, wherein the detected attributes of the combination comprise at least one or more color and shape combinations of one or more active light sources of the one or more traffic lighting devices. [5] Method according to any of the preceding claims, wherein the one or more classifications assigned to the one or more traffic light devices comprise at least a subset of the one or more attributes, wherein the one or more classifications are determined using the fusion head based on at least one combination of the one or more attributes. [6] Method according to any of the foregoing claims, further comprising: Generating one or more component feature vectors corresponding to the one or more traffic light devices depicted in the image, based on at least one or more DNNs processing the image data, where: that one or more attributes are determined using one or more component heads based on at least one application of one or more component feature vectors to one or more component heads, and the one or more properties are determined using the fusion head based on at least one application of a combined feature vector containing a combination of the one or more component feature vectors on which the fusion head is determined. [7] Method according to any of the preceding claims, wherein the one or more component heads comprise at least a first component head and a second component head, wherein the first component head classifies one or more first attributes of the one or more traffic light devices and the second component head classifies one or more second attributes of the one or more traffic light devices. [8] System, comprehensive: one or more processors for: Determining initial data corresponding to one or more initial attributes associated with the traffic light device, based on at least one or more initial layers of a machine learning model processing sensor data obtained using one or more sensors with fields of view or sensor fields encompassing a traffic light device; Determining second data corresponding to one or more second attributes associated with the traffic light device, based on at least one or more second layers of the machine learning model processing the first data; and Performing one or more operations assigned to a machine, based on at least one of the one or more first attributes or the one or more second attributes. [9] System according to claim 8, wherein at least one of the one or more first attributes or of the one or more second attributes comprises at least one of: an alignment of the traffic light device; a housing shape of the traffic light device; active luminous material colours of the traffic light device; active light source forms of the traffic lighting device; a number of light sources in the traffic lighting device; a road user of the traffic light device; or a flashing state of the traffic light device. [10] System according to claim 8 or 9, wherein one or more processors are further configured to: Determining a classification that is assigned to the traffic light device using one or more fusion layers of the machine learning model, where the execution of the one or more operations assigned to the machine is furthermore based at least on the classification. [11] System according to claim 10, wherein the determination of the classification associated with the traffic light device is based on a combination of at least one of: one or more first attributes and one or more second attributes; or a first feature vector and one or more second feature vectors, wherein the first feature vector is used as input for the one or more first layers and the one or more second feature vectors are used as input for the one or more second layers. [12] System according to one of claims 8 to 11, wherein the one or more first attributes determined using the one or more first layers comprise at least one or more color and shape combinations of one or more active light sources of the traffic light device. [13] System according to any one of claims 8 to 12, wherein at least one of the first attributes or the second attributes comprises one or more housing shapes associated with the traffic light device, wherein the one or more housing shapes correspond to at least one of: a vertical case shape; a horizontal housing shape; a doghouse housing shape; or a hybrid pedestrian traffic light housing design. [14] System according to any one of claims 8 to 13, wherein the machine learning model is trained by at least one of: Capturing an image that represents a second traffic light device; Updating one or more parts of the image to produce an updated image that represents the second traffic light device with one or more updated attributes; and Updating one or more parameters associated with one or more first layers or one or more second layers of the machine learning model, based on at least one application of the updated image as a training input to the machine learning model. [15] System according to any one of claims 8 to 14, wherein the system is comprised of at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation processes; a system for carrying out one or more digital twin operations; a system for performing a light transport simulation; a system for the collaborative creation of content for 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system that is implemented using a robot; a system for performing one or more generative AI operations; a system for performing operations using a large language model; a system for performing operations using one or more Vision Language Models (VLMs); a system for carrying out operations using one or more multimodal language models; a system for using or providing one or more inference microservices; a system for carrying out one or more dialogue-oriented AI processes; a system for generating synthetic data; a system for presenting at least virtual reality content, augmented reality content, or mixed reality content; a system that includes one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system that is implemented at least partially using cloud computing resources. [16] One or more processors, comprising: a processing circuit for performing one or more operations corresponding to a machine, based on at least one or more attributes associated with a first traffic light device, wherein the one or more attributes are determined using one or more machine learning models, wherein the one or more machine learning models are trained by at least one of: Obtaining an image that represents a second traffic light device with one or more first attributes; Updating one or more parts of the image to produce an updated image, wherein the updated image represents the second traffic light device with one or more second attributes; and Updating one or more parameters associated with one or more component heads of one or more machine learning models, based on at least one application of the updated image as training input to the one or more machine learning models. [17] One or more processors according to claim 16, wherein updating one or more parts of the image to generate the updated image comprises modifying one or more values ​​of one or more pixels of the image corresponding to one or more active light sources of the second traffic light device, wherein the one or more values ​​are modified such that the one or more active light sources are represented as inactive light sources in the updated image. [18] One or more processors according to claim 16 or 17, wherein the image represents the second traffic light device in a first state, and the updating of one or more parts of the image to generate the updated image comprises updating the image such that the updated image represents the second traffic light device in a second state which differs from the first state. [19] One or more processors according to any one of claims 16 to 18, wherein updating one or more parts of the image to generate the updated image comprises at least one of the following: Updating one or more forms of one or more light sources of the second traffic lighting device; Updating the alignment of the second traffic light device; Updating a housing shape of the second traffic light device; or Updating a number of light sources connected to the second traffic light device. [20] One or more processors according to any one of claims 16 to 19, wherein the one or more processors are contained in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation processes; a system for performing one or more digital twin operations; a system for performing a light transport simulation; a system for collaborative content creation for 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system that is implemented using a robot; a system for performing one or more generative AI operations; a system for performing operations using a large language model; a system for performing operations using one or more Vision Language Models (VLMs); a system for carrying out operations using one or more multimodal language models; a system for using or providing one or more inference microservices; a system for carrying out one or more dialogue-oriented AI processes; a system for generating synthetic data; a system for presenting at least virtual reality content, augmented reality content, or mixed reality content; a system that includes one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system that is implemented at least partially using cloud computing resources.

Images