AUTOMATED VALIDATION AND BENCHMARKING OF PARAMETRIZABLE MODELS IN DISTRIBUTED COMPUTING ENVIRONMENTS

Abstract

Embodiments of the present disclosure relate to the automated optimization and / or evaluation of parameterizable models. In some embodiments, data is recorded or acquired during the optimization of the parameterizable model according to a user instruction. Based on such recording or acquisition, a parameter is then updated during the optimization in some embodiments, for example, through local and / or global optimization. In some embodiments, validation and / or benchmarking is performed based on a type of parameterizable model and one or more performance metrics. In some embodiments, local and / or global validation is performed as part of the validation and / or benchmarking.

Description

BACKGROUND

[0001] A parameterizable model is a mathematical or computational model that contains parameters which can be set or customized to fit specific data or conditions. These parameters are variables within the model that determine its behavior or output, and they can be tuned or modified to better represent the underlying phenomena being modeled. Examples of parameterizable models include statistical models (e.g., regression models) where parameters such as coefficients can be adjusted to fit data; machine learning models (e.g., neural networks) where parameters such as weights and bias can be adjusted during training; sensor models (e.g., radar models) where parameters such as beamwidth, minimum power thresholds, antenna gain, etc., can be optimized; and physical models (e.g., models of physical systems, such as...).mechanical systems, electrical circuits, etc.), where parameters such as mass, speed, and resistance can be set based on experimental data).

[0002] Parameterizable models aim to find an optimal set of parameters (e.g., through training or hyperparameter tuning) to improve their performance and effectiveness in identifying underlying patterns or relationships in the data to which they are applied. This process may involve trying different combinations of parameter values ​​and selecting the most powerful set. However, finding an optimal set of parameters is not only time-consuming but also computationally intensive. Furthermore, there are numerous goals or criteria for optimization, such as real-time operation, model fidelity, and computational efficiency. Optimizing a model to satisfy all these goals simultaneously can increase the complexity of the optimization problem and make finding an optimal solution challenging.

[0003] Parameterizable models must also be evaluated (e.g., validated) to ensure they generalize well to unseen data and to assess their performance under different conditions. Validation refers, for example, to the process of evaluating the model's performance on a separate dataset, distinct from the one used for parameter optimization, to estimate how well the model will perform on new, unseen data. Validation techniques might, for instance, measure a model's accuracy (e.g., classification accuracy), measure how well the model's predictions (e.g., intensity and range) match actual measurements, and / or compare statistical quantities (e.g., different distributions).However, existing validation techniques are not only time-consuming but also difficult to implement, especially in distributed computing environments. For example, parameterizable models can work with large datasets or require significant computing resources for orchestration and validation. SUMMARY

[0004] Embodiments of the present disclosure relate to the automated optimization and / or evaluation (e.g., validation and / or benchmarking) of parameterizable models, in particular parameterizable models used in distributed (e.g., cloud-hosted) computing platforms and / or for generating simulations of scenarios. With regard to optimization, some embodiments first receive a user instruction that defines a sequence of actions associated with the optimization of a parameterizable model. Such a user instruction may, for example, refer to a web user interface selection or a command-line interface command for acquiring or recording data for the optimization of the parameterizable model (e.g.,A command to capture virtual sensor data for light detection and ranging (LIDAR) every 10 meters while a virtual vehicle traverses an environment. In response to receiving such a user instruction, some embodiments then actually record or capture data for optimizing the parameterizable model. Based on such recording or capture, some embodiments then update a parameter during optimization. For example, using the illustration above, one or more of the weights and biases (which in this example represent the "parameter") of the virtual LIDAR sensor model of a neural network are initially initialized to random values ​​or using specific initialization techniques.The virtual LIDAR sensor model then continuously acquires sensor data by simulating the behavior of a real LIDAR sensor according to the user instructions.

[0005] In some embodiments, such optimization or parameter updating includes local optimization. "Local optimization" refers to the process of optimizing one or more parameters of a model on a single compute node (such as a cloud instance, a virtual machine, or a mainframe) or a client within the distributed computing environment of other compute nodes. In some embodiments, such local optimization can be performed locally on each compute node based on a subset of data, and the results are aggregated on a central server for further processing, as described in more detail below.

[0006] In some implementations, parameter optimization or updating includes, additionally or alternatively, global optimization. "Global optimization" refers to the process of aggregating optimization results from multiple nodes or clients in the distributed computing environment to determine the best overall set of parameters for the model being optimized. Accordingly, after each client node has performed local optimization, the server performs global optimization by collecting the results from all nodes, aggregating them, and selecting the best (e.g., average) set of parameters. This aggregated result is then communicated back to all nodes to guide further optimization iterations.

[0007] In some embodiments, validation and / or benchmarking based on a type of parameterizable model and one or more evaluation performance metrics (e.g., key performance indicators, KPIs) is performed for evaluation purposes. For example, in some embodiments, user input is first received specifying the type of parameterizable model. In an illustrative example, a user might select a user interface element corresponding to a type of tree-based model, sensor model, support vector machine (SVM), or the like. Based on the user input specifying the type of parameterizable model, one or more performance metrics (e.g., KPIs) are then calculated in some embodiments. The type of parameterizable model can influence which KPIs are calculated based on various factors such as...are better suited to the inherent properties of the model, the type of problem to be solved, and the desired outcomes. And based on at least the calculation of one or more performance metrics and the user input specifying the type of parameterizable model, some implementations then perform validation or benchmarking of the parameterizable model.

[0008] In some implementations, "local validation" is performed as part of the validation and / or benchmarking process. "Local validation" refers to the process of validating and / or benchmarking a model on a single compute node or client within a distributed computing environment with multiple compute nodes or clients. This validation and / or benchmarking can be performed locally on each node based on a subset of data, and the results are aggregated on a central server for further processing.

[0009] In some implementations, benchmarking and / or model validation additionally or alternatively includes global validation. "Global validation" refers to the process of aggregating validation and / or benchmarking results from multiple nodes or clients in a distributed environment. That is, data from multiple nodes are aggregated on a central server or another node and validated / benchmarked together. Accordingly, after each client node has performed local validation, the server performs global optimization by collecting the results from all nodes and aggregating them.

[0010] The invention is defined by the claims. To illustrate the invention, aspects and embodiments that may or may not fall within the scope of protection of the claims are described herein.

[0011] Embodiments relating to the automated optimization and / or evaluation of parameterizable models are disclosed. In some embodiments, data is recorded or acquired during the optimization of the parameterizable model according to a user instruction. Based on such recording or acquisition, a parameter is then updated during the optimization in some embodiments, for example, through local and / or global optimization. In some embodiments, validation and / or benchmarking is performed based on a type of parameterizable model and one or more performance metrics. In some embodiments, local and / or global validation is performed as part of the validation and / or benchmarking.

[0012] The revelation extends to all novel aspects or features described and / or illustrated here.

[0013] Further features of the disclosure are characterized by the independent and dependent claims.

[0014] Any feature of one aspect of the disclosure can be applied in any suitable combination to other aspects of the disclosure. In particular, procedural aspects can be applied to apparatus or system aspects, and vice versa.

[0015] Furthermore, features implemented in hardware can be implemented in software and vice versa. Any reference to software and hardware features here should be interpreted accordingly.

[0016] Each system or device feature described here can also be provided as a process feature, and vice versa. System and / or device aspects that are functionally described (including means plus functional features) can alternatively be expressed in terms of their corresponding structure, such as a suitably programmed processor and associated working memory.

[0017] It is also understood that certain combinations of the various features described and defined in each aspect of the revelation can be implemented and / or provided and / or used independently of one another.

[0018] The disclosure also provides computer programs and computer program products comprising software code designed to perform one of the methods described herein when executed on a data processing device and / or to embody one of the device and system features described herein, including one or all component steps of a method.

[0019] The disclosure also includes a computer or computing system (including networked or distributed systems) with an operating system that supports a computer program for carrying out the procedures described herein and / or for embodying the device or system features described herein.

[0020] The disclosure also provides a computer-readable medium on which one or more of the aforementioned computer programs are stored.

[0021] The revelation also provides a signal that carries one or more of the aforementioned computer programs.

[0022] The disclosure extends to methods and / or devices and / or systems as described herein with reference to the accompanying drawings.

[0023] Aspects and embodiments of the disclosure will now be described purely by way of example with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The systems and methods presented here for sensor simulation and sensor model learning using generative machine learning are described in detail below with reference to the accompanying drawings. These show: Fig. 1 an exemplary pipeline for the automatic optimization and evaluation of a parameterizable model, according to some embodiments of the present disclosure; Fig. 2 a block diagram illustrating how various tasks are performed by local and global optimization and evaluation, according to some embodiments of the present disclosure; Fig. 3 a block diagram of a distributed system architecture, according to some embodiments of the present disclosure; Fig. 4 a sequence diagram illustrating how a user device, a server and client compute nodes work together to perform local and global optimization, according to some embodiments of the present disclosure; Fig. 5 a sequence diagram illustrating how a user device, a server and client compute nodes work together to perform a local and global evaluation (benchmarking and / or validation), according to some embodiments of the present disclosure; Fig. 6 a flowchart of an exemplary procedure for optimizing a parameterizable model, according to some embodiments of the present disclosure; Fig. 7 a flowchart of an exemplary procedure for validating and / or benchmarking a parameterizable model, according to some embodiments of the present disclosure; Fig. 8 a visualization of KPI values ​​and parameter values ​​via optimization iterations, according to some embodiments of the present disclosure; Fig. 9A an illustration of an exemplary autonomous vehicle, according to some embodiments of the present disclosure; Fig. 9B is an example of camera locations and fields of view for the exemplary autonomous vehicle from Fig. 9A, according to some embodiments of the present disclosure; Fig. 9C a block diagram of an exemplary system architecture for the exemplary autonomous vehicle from Fig. 9A, according to some embodiments of the present disclosure; Fig. 9D is a system diagram for the communication between one or more cloud-based servers and the example autonomous vehicle. Fig. 9A, according to some embodiments of the present disclosure; Fig. 10 a block diagram of an exemplary computing device suitable for use in the implementation of some embodiments of the present disclosure; and Fig. 11 a block diagram of an exemplary data center suitable for use in the implementation of some embodiments of the present disclosure. DETAILED DESCRIPTION

[0025] Some embodiments relate to the automated optimization and / or evaluation (e.g., validation and / or benchmarking) of parameterizable models. With regard to optimization, some embodiments first receive a user instruction that defines a sequence of actions associated with optimizing a parameterizable model. Such a user instruction might, for example, point to a web user interface selection or a command-line interface command for acquiring or recording data for optimizing the parameterizable model. In an illustrative example, the user instruction might be a command to acquire virtual sensor data for light detection and distance measurement (LIDAR) every 10 meters as a virtual vehicle (or other ego-machine) traverses an environment.Another action in the user instruction can be to move the target distance when acquiring data from 10 meters to 20 meters, so that measurements take place at different distances.

[0026] In response to receiving such a user instruction, some embodiments actually record or capture data during the optimization of the parameterizable model. For example, using the illustration above, in some embodiments the virtual sensor model's data is first captured at 10 meters and then at 20 meters. Based on such a recording or capture, in some embodiments a parameter is then updated during the optimization. For example, using the illustration above, one or more of the weights and biases (which in this example represent the "parameter") of a neural network's virtual LiDAR sensor model are initially initialized to random values ​​or using specific initialization techniques.The virtual LIDAR sensor model then continuously acquires sensor data by simulating the behavior of a real LIDAR sensor according to the user instructions.

[0027] The acquired sensor data can be fed into the model for prediction. Continuing this example, the model uses the current weights and bias to generate predictions about the environment based on the input sensor data. The model's predictions can then be compared to the ground truth or expected results. The error between the predicted values ​​and the virtual sensor data can then be calculated using a loss function (e.g., mean squared error, "MSE"). In some embodiments, the gradient of the loss function is calculated with respect to the model's parameters (e.g., its weights and bias). This gradient represents the direction and magnitude of the change required to reduce the error. The weights and bias can then be updated using an optimization algorithm (e.g., gradient descent).Such an update function adjusts the parameters in a direction that minimizes the loss function based on the calculated gradients. For example, during gradient descent, the weights and bias are updated by subtracting a fraction of the gradient from their current values, scaled by a learning rate hyperparameter. This process of data acquisition (according to user instructions), prediction, error calculation, gradient computation, and parameter update continues as the virtual LiDAR sensor model acquires more data. With each iteration, the model's weights and bias are refined to better identify the underlying patterns in the sensor data and improve the accuracy of the predictions.

[0028] In some embodiments, such optimization or parameter updating includes local optimization. "Local optimization" refers to the process of optimizing one or more parameters of a model on a single compute node (e.g., a back-end server or cloud compute node) or a client within the distributed computing environment of other compute nodes. In some embodiments, such local optimization can be performed locally on each compute node based on a subset of data, and the results are aggregated on a central server for further processing, as described in more detail below. In some embodiments, each node has a copy of the same model, with parameters that may differ. Multiple tasks may run on a single node, or multiple nodes may participate in a single task.In some embodiments, a task corresponds to a local optimizer or a global optimizer.

[0029] In some implementations, local optimization involves calculating one or more performance metrics (e.g., key performance indicators (KPIs)) and then transferring these and the updated parameters to another compute node (e.g., to prepare for global optimization, as described below). In an illustrative example, a client app could be a software application running on a smartphone that controls a camera-equipped drone for aerial photography. The model parameter could be the camera's exposure time.

[0030] When the client app receives a signal from the user to start a scenario step (e.g., capturing images during flight), it can update the camera's exposure time parameter to optimize image quality for the current lighting conditions. The model task, running on the drone's onboard computer, can then receive the signal from the client app to begin capturing images. It initiates the scenario step by activating the camera and capturing images of the landscape below. After image capture is complete, the model task processes the recorded images and communicates the data format (e.g., image resolution, file format) to the client app. It can also set a flag indicating that recording is complete.The local optimizer in the client app can then evaluate the key performance indicators (KPIs) of the captured images based on factors such as brightness, sharpness, and contrast. The camera's exposure time parameter can then be adjusted to optimize these metrics for future image captures. The client app can then send log data containing the calculated KPI values ​​and updated parameter settings to a server app for further analysis and monitoring. This enables real-time tracking of the optimization progress. Such an optimization process can be iterative, with the client app and the model task repeating the steps described above for a predetermined number of optimization steps or until a specified optimization tolerance level is reached.

[0031] In some implementations, parameter optimization or updating includes, additionally or alternatively, global optimization. "Global optimization" refers to the process of aggregating optimization results from multiple nodes or clients in the distributed environment to determine the best overall set of parameters for the model being optimized. Accordingly, after each client node has performed local optimization, the server performs global optimization by collecting the results from all nodes, aggregating them, and selecting the best (e.g., mean) parameter set. This aggregated result is then communicated back to all nodes to guide further optimization iterations. In some implementations, each local node receives the same best parameter set (e.g., a mean-like strategy).In other embodiments, each local node receives a new or separate parameter calculated from the aggregated results. In some embodiments, the global optimization also includes a global optimization strategy to optimally trigger new steps for local optimization. For example, a Basin-hopping style optimization strategy can be used. Basin-hopping is a global optimization algorithm used to find the global minimum of a function, which can be particularly useful for problems with many local minima. The technique combines local optimization with a random search of the solution space, making it possible to bypass local minima and explore the global landscape more effectively.

[0032] Such global optimization can, in some implementations, incorporate federated learning. Federated learning involves training a machine learning model on multiple decentralized devices or servers, each with its own local data samples, without sharing them centrally. Instead of sending raw data to a central server, local models are trained locally, and only model updates (e.g., gradients) are exchanged with a central server or between other nodes. This approach ensures data privacy and security, as sensitive data remains on the local device or server.

[0033] In some embodiments, the global optimization includes the following operations. Each compute node in a multi-node system can transmit its updated parameters to a first compute node (e.g., a central server). In some embodiments, one or more of these parameters can then be modified based on aggregated parameter values ​​from at least the first compute node. In an illustrative example, each node can send its learning rate (also called the "model rate") and / or updated weights to a central server. The central server can then aggregate each of the updated parameter values ​​from each compute node, for example, by calculating the average of each received weight / learning rate. The central server can then send the average to all other compute nodes, which serves as a new starting point (e.g., for another epoch / round of training).

[0034] In some embodiments, the parameterizable model is additionally or alternatively evaluated by validating and / or benchmarking it. "Benchmarking" refers to comparing or evaluating the performance (e.g., accuracy) of the parameterizable model against a standard or reference point (i.e., a "benchmark"). The purpose of using benchmarks is to provide a basis for evaluating model performance. As an illustrative example, with regard to natural language processing (NLP) models, in some embodiments the model's performance can be evaluated against standard NLP benchmarks such as the Stanford Sentiment Treebank (sentiment analysis), SQuAD (question answering), or GLUE (General Language Understanding Evaluation).

[0035] In some embodiments, validation and / or benchmarking is performed based on a type of parameterizable model and one or more performance metrics. For example, in some embodiments, user input is first received specifying the type of parameterizable model. In an illustrative example, a user can select a user interface element corresponding to a type of tree-based model, sensor model, support vector machine (SVM), neural network, ensemble method, Markov model, Bayesian model, or the like. Based on the user input specifying the type of parameterizable model, one or more performance metrics (e.g., KPIs) are then calculated in some embodiments. The type of parameterizable model can influence which KPIs are calculated based on various factors, such as...The inherent properties of the model, the type of problem to be solved, and the desired consequences are better suited. For example, a key-value pair data structure (such as a lookup table or a hash table) can contain keys that specify the model type and corresponding values ​​that specify the respective KPIs. Linear models (such as linear regression, logistic regression), for instance, can be specified as keys, and since these models are often used for regression or binary classification tasks, KPIs such as mean absolute error (MAE), mean squared error (MSE), and / or R² can be listed as values ​​in the same entry or record.Accordingly, in some embodiments, for example, the user input specifying the model type is used as input to search for the corresponding key indicating the model type (e.g., via fuzzy matching) in order to return the KPIs that need to be calculated. And based on at least the calculation of one or more performance metrics and the user input specifying the type of parameterizable model, some embodiments then perform a validation or benchmarking of the parameterizable model.

[0036] In some implementations, "local validation" is performed as part of the validation and / or benchmarking process. "Local validation" refers to the process of validating and / or benchmarking a model on a single compute node or client within a distributed computing environment with multiple compute nodes or clients. Similar to local optimization, this validation and / or benchmarking can be performed locally on each node based on a subset of data, and the results are aggregated on a central server for further processing, as described in more detail below.

[0037] In some embodiments, the local validation and / or benchmarking includes the following operations. The user input described above may further define a sequence of actions that constitute a capture or recording task associated with the validation and / or benchmarking. Based at least on the user input defining the sequence of actions, data is recorded or captured in some embodiments. Using the illustration above, the user input may, for example, include a command to capture virtual sensor data for light detection and distance measurement (LIDAR) every 10 meters as a virtual vehicle (or other ego-machine) traverses an environment.Based at least in part on calculating one or more performance metrics and recording or capturing the data according to user input, in some embodiments the one or more performance metrics (e.g., and / or updated parameters for optimization to prepare for global validation, as described in more detail below) are then transferred to a first compute node of several compute nodes.

[0038] In some implementations, benchmarking and / or model validation additionally or alternatively includes global validation. "Global validation" refers to the process of aggregating validation and / or benchmarking results from multiple nodes or clients in a distributed environment. That is, data from multiple nodes are aggregated on a central server or another node and validated / benchmarked together. Accordingly, after each client node has performed local validation, the server performs global optimization by collecting the results from all nodes and aggregating them.

[0039] Accordingly, in response to validation or benchmarking of a parameterizable model on the first node, the first node transmits one or more performance metrics to a second compute node (e.g., a central server). This allows the second node to perform a global validation, based at least on the second compute node's data, using aggregated performance metrics from all other compute nodes in the distributed environment. For example, each node in the distributed environment can validate a different KPI and / or a different scenario (e.g., as defined in the user input). The central server can then receive these different KPIs from all nodes and, in response, aggregate the results (e.g., by summarizing all results in a report and / or generating overall statistics on the quality of the aggregated dataset).In other words, each compute node can have its own model that needs to be validated / benchmarked. Since each model (even if it's the same type of model) can have different KPIs, input data, output data, and / or parameters on each compute node, its numerical values ​​for validation and / or benchmarking may differ. Therefore, it can be useful to determine how the model performs using different KPIs or other data. This validation process could involve checking for consistency, accuracy, completeness, or other criteria relevant to data integrity and quality.

[0040] The systems and procedures described here can be used without restriction by non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more adaptive driver assistance systems (ADAS)), steered and unsteered robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled with one or more trailers, hydrofoils, boats, shuttles, emergency vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, trains, underwater vehicles, remotely controlled vehicles such as drones, and / or other vehicle types.Furthermore, the systems and methods described here can be used for a variety of purposes, including but not limited to machine control, machine locomotion, machine propulsion, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and monitoring, simulation and digital twinning, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital twinning, data center processing, conversational AI, light transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation for 3D assets, using language models such as large language models (LLMs) or vision language models (VLMs), cloud computing, generative AI, and / or other suitable applications.

[0041] The disclosed embodiments can comprise a variety of different systems, such as automotive systems (e.g.a control system for an autonomous or semi-autonomous machine, a perception system for an autonomous or semi-autonomous machine, systems implemented using a robot, aviation systems, media systems, boat systems, intelligent area surveillance systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using an edge device, systems that include one or more virtual machines (VMs), systems for performing operations to generate synthetic data, systems that are at least partially implemented in a data center, systems for performing conversational AI operations, systems for implementing one or more language models, such as...one or more large language models (LLMs) or one or more vision language models (VLMs), systems for performing light transport simulation, systems for performing collaborative content creation for 3D assets, systems that are implemented at least partially using cloud computing resources, and / or other types of systems. EXEMPLARY PIPELINE

[0042] With reference to Fig. 1 is Fig. 1 An exemplary pipeline 100 (referred to as "Pipeline 100") for automatically optimizing and evaluating a parameterizable model, according to some embodiments of the present disclosure. It should be noted that this and other arrangements described herein are presented only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, arrays, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as single or distributed components, or in conjunction with other components, in any suitable combination and at any suitable location. Various functions described herein that are performed by entities may be executed by hardware, firmware, and / or software.Various functions can be performed, for example, by a processor that executes instructions stored in main memory. In some embodiments, the systems, methods, and processes described here can be implemented using similar components, features, and / or functionality to that of the exemplary autonomous vehicle 900. Fig. 9A-9D, the exemplary computing device 1000 from Fig. 10 and / or the exemplary data center 1100 from Fig. 11 will be executed.

[0043] Generally speaking, Pipeline 100 is responsible for optimizing and / or evaluating one or more parameterizable models. Pipeline 100 includes a user interface component 102, input data 104, a gateway API 106, an optimization / evaluation scheduler and executor 108, a node job handler 110, tasks 112, 114, and 116, storage 120, a federated optimization dashboard 122, a federated optimization service 124, a job description 126, and dashboard information 130.

[0044] Fig. Figure 1 illustrates an embodiment in which automatic optimization and evaluation take place across various compute nodes based on the input of user input 104. At an initial point in time, the user interface component 102 (e.g., a browser or a command-line interface (CLI)) receives the user input 104. The user input 104 contains a scenario, one or more key performance indicators (KPIs), input data, and configuration data. A “scenario,” as described herein, refers to a user instruction that defines a sequence of actions associated with the optimization and / or evaluation of a parameterizable model.In an illustrative example, a user can execute a scenario script via a CLI that outlines a sequence of actions the self-driving car model must perform, along with the parameters to be set and the data to be recorded at each step, as follows: Step 1 - the simulated self-driving car starts in an urban environment, where the parameters are speed limit, acceleration profile, and distance to the nearest obstacle. The recorded data includes virtual vehicle speed, distance traveled, and obstacles detected.

[0045] In step 2 (another action in the sequence), the car navigates a lane change maneuver, with parameters including lane change speed, lane change distance, and turn signal activation time. Recorded data can include the success of the lane change, the time required for the lane change, and the distance to nearby vehicles. In step 3, the car approaches and navigates through an intersection, with parameters including approach speed, stopping distance, and decision threshold. Recorded data can include intersection crossing time, successful lane selection, and traffic light detection.

[0046] In step 4, the user specifies that the car should find and park in an available parking space, with parameters including the sensitivity of the parking space detection, the precision of the parking maneuver, and the distance to the curb. Recorded data can include the parking success, the time required for parking, and the proximity to other vehicles or obstacles.

[0047] The "user data" for input 104 can be any suitable data, such as the type of model the user will optimize and evaluate, a sensor profile, or similar information. A sensor profile refers to a set of parameters or properties that describe the behavior and characteristics of a sensor. Sensor profiles can be used in modeling and simulation to represent the response of different sensors to various stimuli or inputs. These parameters can include sensitivity, noise characteristics, resolution, dynamic range, response time, and other relevant properties that define how the sensor interacts with its environment.For example, in relation to the scenario illustrated above, the user can provide a sensor profile for a virtual LIDAR model in 4 steps, which includes range, angular resolution, scan rate, accuracy, field of view (FOV), operating wavelength, and noise characteristics.

[0048] The configuration data within input 104 contains all the appropriate settings and specifications that the user can define to customize the behavior and parameters of the evaluation and optimization process within the framework. These configurations help tailor the framework's operation to specific requirements and preferences. For example, one configuration includes the number of optimization steps. This specifies the maximum number of steps or iterations to be performed during the optimization process. This can help control the duration or complexity of the optimization procedure. Another user-specified configuration is the number of cluster nodes.In other words, the user can specify the number of nodes in a cluster to be used for distributed processing, potentially speeding up the optimization (and / or evaluation) process by parallelizing computations across multiple nodes.

[0049] Another configuration option is to include parameter limits. These define the ranges or constraints for the parameters subject to optimization. Setting parameter limits helps the optimization algorithm search within a meaningful range of values. Consider an exemplary scenario where, in some embodiments, the parameters of a machine learning model for image classification tasks are optimized. In this context, the parameter limits could refer to the acceptable range or constraints for the values ​​of certain parameters within the model. For example, in some embodiments, the learning rate parameter for a neural network model can be optimized using a gradient descent-based optimization algorithm. The learning rate determines the step size in each iteration of the optimization process.If the learning rate is too low, the optimization process can be too slow. If the learning rate is too high, the process can oscillate or diverge. In this example, the parameter limits for the learning rate could be defined as follows: minimum learning rate: 0.001 and maximum learning rate: 0.1. This means that during the optimization process, the algorithm only considers learning rates within the range of 0.001 to 0.1. Values ​​outside this range are not explored. Setting parameter limits in this way helps the optimization algorithm search within a reasonable and meaningful range of values, preventing it from trying impractical or invalid values.Parameter limits ensure that the optimization process remains efficient and effective by restricting the search space to values ​​that are likely to produce good results, thus avoiding unnecessary searching of irrelevant or harmful parameter configurations.

[0050] Another example of user-specific configuration involves the workflow. The workflow specifies whether the framework should perform validation, optimization, or both. This allows the user to choose the desired workflow based on their current goals.

[0051] After the user specifies one or more of these data points in input 104, the input is then forwarded to the Gateway API 106. The Gateway API 106 refers to an intermediary layer that lies between clients (e.g., client applications containing the user interface component 102) and backend services or systems, such as components 108, 110, and 124. The Gateway API 106 thus acts as a single point of entry for accessing multiple services.

[0052] The job scheduler service 108 (e.g., NVIDIA's FARM MICROSERVICE) is generally responsible for scheduling the jobs specified in user input 104, taking into account various compute nodes. In some implementations, the job scheduler service 108, via the node job handler 110 (e.g., FARM JOBS MICROSERVICE), enables efficient resource utilization by sharing compute resources across multiple machines or compute nodes. This ensures optimal performance and scalability. For example, users submit a scenario specified in tasks 112, 114, and / or 116 to a farm queue (e.g., the node job handler 110). The farm queue manages the task queue and assigns it to available farm agents. Each farm agent runs on a specific compute node (e.g., a machine). Thus, there can be multiple farm agents, each running on a given compute node.Farm agents continuously monitor the farm queue for pending tasks. When a task (e.g., 112) becomes available, the farm agent retrieves it for execution. Farm agents allocate resources (CPU, GPU, memory) based on the task's requirements. If a node has multiple GPUs, the farm agent can distribute tasks across them. The farm agent then executes the task on the assigned node.

[0053] A “task,” as described herein, refers to a specific computational or data processing activity performed within the framework for the automatic validation, benchmarking, and / or optimization of parameterizable models. These tasks are fundamental components of the workflow orchestrated by the framework and serve various purposes, such as model training, parameter optimization, data validation, benchmarking, or the orchestration of validation workflows. For example, a parameter optimization task focuses on finding the optimal values ​​for a model’s parameters to maximize its performance. This may involve iterative optimization algorithms, such as gradient descent or genetic algorithms.On the other hand, in some implementations, a data validation task validates a model's performance against a set of key performance indicators (KPIs) using recorded data. This ensures that the model behaves as expected and meets predefined quality standards. A benchmarking task compares the performance of a given model against standard benchmarks or reference models. This helps assess the model's effectiveness and competitiveness compared to existing solutions.

[0054] Upon completion, the task result is sent back to the farm queue. Resource sharing ensures efficient use of available nodes. If a node is busy, the farm queue or node job handler reassigns tasks to less busy nodes. Multiple users can submit tasks to the farm queue. Farm agents process tasks from different users concurrently. Compute nodes can be dynamically added or removed. For example, additional nodes can be added during peak rendering times to speed up the process. In some implementations, the job scheduler is infrastructure-agnostic. This allows it to run on workstations, physical servers, cloud platforms like Kubernetes, or any other architecture.

[0055] In Fig. Task 112 can be scheduled to run on compute node 1. Task 114 can be scheduled to run on compute node 2. And task 116 can be scheduled to run on compute node 3. After scheduling by the job scheduler service 108, the tasks are consolidated via aggregate output 118 and stored in storage 120 (e.g., random access memory (RAM), hard disk, or storage array, such as a storage area network (SAN)).

[0056] The federated optimization service 124 (e.g., EVOLVE MICROSERVICE) then accesses the aggregated output 118 to generate a federated job description 126. The federated optimization service 124 is generally responsible for generating a description that specifies how one or more compute nodes will optimize parameters during the received tasks. For example, the federated optimization service 124 can generate the federated description 126, which describes how federated learning occurs through the process of initialization, distribution of the initial model parameters to the participating compute nodes, local model training, model updates, and global model update bias, and the like, as further described below. The federated optimization dashboard service 122 (e.g., EVOLVE DASHBOARD MICROSERVICE) is generally responsible for the presentation (e.g.,to generate or initiate (on an administrator's user device) optimization plans and KPI values ​​as determined by the Federated Optimization Service 124.

[0057] In various embodiments, the federated job description 126 is then returned via the gateway API 106, which is then forwarded to the job scheduler service 108 and the node job handler 110, allowing these components to modify the jobs or tasks to be performed, depending on how the federated optimization service 124 instructs different nodes to participate in the optimization. The gateway API 106 is also responsible for generating the dashboard information 130, which is then returned to the user interface component 102, enabling a user device to display relevant information, such as real-time values ​​as they occur throughout the optimization and / or evaluation process (e.g., parameter values, such as weights, and how they change over different epochs).For example, the dashboard information can contain 130 KPI values ​​across different time slices or epochs, sensor data across different time slices, parameter values ​​across different time slices or epochs, scenario data, and the like.

[0058] Fig. Figure 2 is a block diagram illustrating how different tasks are performed through local and global optimization and evaluation, according to some embodiments. In some embodiments, the same task can run on a single compute node or on multiple compute nodes, depending on how the job scheduler service 108 and the node job handler 110 have scheduled the jobs. In some embodiments, each task 201, 203, 205, 207, and 209 represents modules and functionality that are performed on a separate compute node among multiple compute nodes in a distributed computing environment.

[0059] Fig. 2 contains different tasks 201 (which includes, for example, task 112 of Fig. 1), 203, 205, 207, and 209. Each task contains a specific image, such as a federated image, and other data, or is associated with one. For example, task 201 contains a federated image 211 (e.g., an EVOLVE IMAGE). In some embodiments, each federated image described herein is a Docker container responsible for federated optimization. A “Docker container,” as described herein, is a lightweight, self-contained, executable package containing everything needed to run a piece of software, including the code, runtime, system tools, libraries, and settings. It is built from a Docker image, which is essentially a snapshot of a file system (e.g., the federated optimization service 124) containing the necessary configuration to run an application.Containers share the kernel and resources of the host system (e.g., a compute node), which reduces overhead compared to traditional virtual machines. Multiple Docker containers can run on a single host.

[0060] The Server-App-Container 211 is generally responsible for performing global optimization and global validation, as described herein. For example, the Server-App-Container 211 can aggregate optimal parameters and results from each compute node and send an average of the optimal parameters to the different compute nodes in a distributed environment.

[0061] Task 203 contains the federated image 213, which contains the client app container 213-1. The client app container 213-1 is generally responsible for performing local optimization and / or validation, as described herein, and for starting the model image 215 (e.g., a model container). After running these local operations, the client app container passes its updated parameter values ​​and / or log data to the server app container 211-1, as described in Fig. Figure 2 illustrates this. Task 205 contains a model image 215, which contains the model task / data record 215-1. A "model image" refers to a pre-packaged representation of a parameterized model that encapsulates not only the model architecture but also its parameters, dependencies, and configurations in a standardized and portable format. A model image bundles all the necessary components of a model, including the model architecture, the trained parameters, the preprocessing steps, and any additional dependencies required for inference, just as Docker images encapsulate the code, dependencies, and configurations of an application. Just as Docker images can be deployed and run consistently in different environments, model images ensure the portability of machine learning models across different platforms, compute nodes, and infrastructures.This enables the seamless deployment and execution of models in diverse computing environments, from local machines to cloud-based servers. In some implementations, such model images allow for the scalable deployment of models by providing a lightweight and efficient representation that can be easily deployed and scaled across multiple compute nodes of a distributed system.

[0062] The model task / data recording module 215-1 performs the actual recording and / or acquisition tasks specified by the user in input 104. Such recorded data (e.g., sensor readings from a virtual LiDAR) is forwarded to the client app container 213-1, which then optimizes and / or validates the recorded data, for example, in Fig. 2 illustrated.

[0063] Task 207 contains its own federated image 217 and runs its own client app container 217-1. Client app container 217-1 also performs local optimization and / or validation and starts the model image 220. The model task / data recorder 220 sends its data, or the recorder data, to client app container 217-1, as shown in Fig. Figure 2 illustrates this. The client app container 217-1 also sends its parameters / log data to the server app container 211-1.

[0064] Fig. Figure 3 is a block diagram of a distributed system architecture 300 (referred to as "System 300"), according to some embodiments. In some embodiments, one or more components in Fig. 1 and / or Fig. 2 represents one or more components of the system 300. For example, in some embodiments, the compute node 304 of the central server represents the functionality of the server application container 211-1 of Fig. 2 and / or the federated optimization service 124 of Fig. 1 represents or contains these. In some embodiments, each of the client compute nodes 320, 330, and 340 represents or hosts a specific client application container, such as 213-1 or 217-1. In some embodiments, each of the model images 326, 336, and 346 represents the model image 215 or 220 of Fig. 2. It is understood that, although System 300 is illustrated with a specific type and number of compute nodes, any quantity and any type are possible. For example, there may be more client compute nodes and more central server compute nodes. In some embodiments, the central server compute node 304 and one or more of the client compute nodes 320, 330, and / or 340 represent logic or software modules residing on the same physical hardware host. In other words, a client and a server may be hosted on the same compute device. In other embodiments, each of the compute nodes represents individual and separate hardware compute devices (e.g., individual cloud nodes in a cloud computing architecture).

[0065] Distributed system architecture (300) is a computing paradigm in which multiple independent computing devices (e.g., "compute nodes") work together as a unified system to perform a task or execute a program. Unlike traditional centralized systems, where all processing takes place on a single machine, distributed systems distribute the workload across multiple machines connected via one or more networks (310). This architecture offers several advantages, including improved scalability, fault tolerance, and performance. In a distributed system, each individual computer or instance is referred to as a "compute node." These compute nodes can differ in their hardware specifications, operating systems, and geographic locations. Compute nodes communicate with each other over a network to work together on tasks (e.g.,(on tasks 112, 114 and 116) and to share resources. In some embodiments, compute nodes 304, 320, 330 and 340 constitute a "cluster", which is a group of interconnected compute nodes that work together to perform computational tasks.

[0066] Initially, a user formulates the scenario instruction 303 (and / or any input data 104) in a CLI or browser on the user device 302 (e.g., a mobile phone, a desktop computer, etc.). The central server's compute node 304 then takes the scenario instruction 303 as input and, in response, sends an initial parameter to each client compute node—computation nodes 320, 330, and 340. Each client compute node then performs local optimization and evaluation of its respective model images (as described in more detail below) and forwards its optimization and / or evaluation data (e.g., the parameter / log data, as referenced in the following). Fig. 2 described) back to the central server's compute node 304, so that the central server's compute node 304 can perform global optimization (e.g., federated learning) via the global optimization module 306 and global evaluation (e.g., benchmarking and / or validation), as described in more detail below.

[0067] In some embodiments, some or each of the compute nodes 304, 320, 330, and 340 represent separate hardware hosts, such as different computers. Alternatively, in some embodiments, some or each compute node represents a virtual instance, such as a virtual machine or a container (one or more of which are hosted on a single hardware host).

[0068] Each local optimization module in 322, 332, and 342 is generally responsible for performing local optimization, as described in more detail below. Each of the local evaluation modules 324, 334, and 344 is responsible for performing local evaluation (e.g., local benchmarking and / or local validation), as described in more detail below. In some embodiments, the model images 326, 336, and 346 together represent an image of the same parameterizable model (e.g., a sensor-based model that predicts LiDAR data), except that each of the model images is initialized or initially contains different parameters and / or different KPIs are used for validation and / or benchmarking of the respective models. Alternatively, in some embodiments, the model images 326, 336, and 346 represent different layers of the same model.For example, model image 326 can contain various input and intermediate layers, such as convolutional and max-pooling layers of a CNN (and no output layers), while model image 336 can contain one output layer, such as a dense layer (e.g., a fully connected layer) of a CNN. A dense layer is a type of layer that is fully connected, meaning that every neuron in the dense layer is connected to every neuron in the previous layer. Each connection between neurons has a learned weight and an associated bias.

[0069] In some embodiments, communication between compute nodes occurs via message transfer protocols, such as TCP / IP, UDP, or HTTP, over one or more networks 310. The compute nodes can send messages to each other to exchange data, coordinate activities, and synchronize their operations. Communication can be synchronous or asynchronous, depending on the requirements of the system 300.

[0070] In some embodiments, the System 300 enables the sharing of resources, such as processing power, main memory, storage, and input / output devices (e.g., virtual and / or hardware-based), across multiple compute nodes. This allows for efficient resource utilization and facilitates collaboration between nodes to accomplish tasks that may require more resources than a single node can provide.

[0071] One advantage of distributed systems is scalability. By adding more compute nodes to the System 300, its capacity can be increased to handle larger workloads and accommodate more users. Distributed systems can be scaled horizontally by adding more compute nodes, vertically by upgrading existing nodes, or both.

[0072] In some embodiments, System 300 is fault-tolerant, as it is designed to be resistant to failures. Because System 300 contains multiple independent nodes, the failure of one compute node does not necessarily lead to an interruption of the entire System 300. Fault tolerance mechanisms, such as replication, redundancy, and distributed consensus protocols, help ensure that System 300 remains operational even in the presence of failures.

[0073] In some embodiments, the system includes 300 load balancing algorithms to distribute incoming requests or tasks evenly across multiple compute nodes, preventing any single node from becoming overloaded. This helps maximize resource utilization, minimize response times, and improve system performance.

[0074] Fig. Figure 4 is a sequence diagram illustrating how a user device, a server, and client compute nodes work together to perform local and global optimization, according to some embodiments. In some embodiments, the user device 402 represents the user device 302 of Fig. 3. In some embodiments, server 404 represents the compute node 304 of the central server of Fig. 3. In some embodiments, the client compute nodes 320, 330 and 340 are of Fig. 3 in the client compute node 406 of Fig. 4. In some embodiments, both the server 404 and a client, such as one of the client compute nodes 406, can run on the same compute device. In some embodiments, the server 404 is located on a different compute node with respect to the other client compute nodes 406.

[0075] In step 1, the user device transmits a scenario, one or more initial parameters, and configuration data (e.g., number of steps to be performed for optimization, number of compute nodes to accelerate optimization, parameter limits, and workflow type (e.g., optimization or validation)) to the 404 server. In some embodiments, a client application of the 402 user device initially includes a data loader that first formats the user scenario, one or more initial parameters, and configuration data. A "data loader," as described herein, reads recorded data into a known format. A data loader is responsible for loading and preparing datasets for use in training, validating, or testing models.Data loaders are essential for handling various data formats, preprocessing data, and efficiently feeding data into models during training or evaluation. For example, input 104 might contain a dataset of images representing handwritten digits (0-9), known as the MNIST dataset. Each image is represented as a matrix of pixel values, and each image has a corresponding label indicating the digit it represents. The MNIST data loader performs data loading, data transformation, batch processing, shuffling, and iteration.

[0076] The data loader first reads the image files and their associated labels from the hard drive into memory. This ensures that the images and labels are properly paired and organized for further processing. Since neural networks typically expect input data in a specific format, the data loader performs all necessary transformations on the raw data. For example, it can scale the images to a standard size, normalize the pixel values ​​to a specific range (e.g., 0 to 1), and convert the data into tensors, which are the fundamental data structure used by many deep learning frameworks. To improve efficiency and leverage parallelism during training, in some implementations the data loader divides the dataset into smaller batches of images and labels. Each batch contains a fixed number of samples, such as...16 to 128 or more, depending on hardware limitations and the size of the dataset. To prevent the model from learning specific patterns related to the order of the data, in some embodiments the data loader randomly shuffles the stacks before each epoch (a complete pass through the dataset). This ensures that the model is exposed to data samples in different orders during training, leading to better generalization. In some embodiments, the data loader provides an iterator interface that allows the training algorithm to iterate through the data stacks sequentially. In each iteration, the training algorithm processes a stack of input data, computes the corresponding predictions using the model, calculates the loss (error), and updates the model parameters through backpropagation.

[0077] In step 2, the 404 server then transmits the one or more initial parameters and the scenario to each of the client compute nodes 406. In some embodiments, the 404 server transmits different / unique initial parameters to each client compute node. For example, if multiple parameters are specified by the user, the 404 server transmits a first parameter to the first client compute node 406 and a second parameter to the second client compute node 406. In these embodiments, each node could search in a different parameter space. Accordingly, some embodiments attempt to determine the "best" parameter by distributing the work / parameters among the client compute nodes.

[0078] In other embodiments, the same initial parameters are sent to all client compute nodes 406. This ensures that each client compute node starts with the same initial parameter values ​​before local optimization tasks are performed. For example, the server 404 distributes an initial set of neural network parameters (weights and biases) to all client compute nodes 406 involved in the optimization task. Each client compute node receives the same initial parameter values ​​from the server 404. The client compute nodes 406 use these initial parameters to initialize their local neural network models before the optimization process begins.During the optimization process, each client compute node independently updates its local model parameters based on its subset of training data and communicates the updated parameters and performance metrics back to the 404 server for aggregation and analysis. By sending the same initial parameter set to each client compute node, the 404 server ensures the consistency and synchronization of the optimization process across all 406 client compute nodes, enabling collaborative optimization toward a common goal.

[0079] In steps 3 through 6, local optimization takes place on each client compute node (e.g., as performed by the local optimization modules 322, 332, and 342). In step 3, each client application (e.g., the client app container 213-1) on each client compute node in 406 sets up a local optimizer and initializes a model task (e.g., DRIVESIM). For example, in some embodiments, the parameters of a machine learning model are optimized. This model uses data acquired by driving simulation software to predict vehicle trajectories in various driving scenarios. The client application running on a compute node sets up a local optimization algorithm, such as gradient descent, to adjust the model parameters. This initializes the model task by configuring the driving simulation software to simulate different driving scenarios.

[0080] In step 4, each client application on each client compute node records data according to the scenario provided by the user. For example, the client application first updates one or more initial parameters and marks the start of the scenario containing the data recording. Each client application sets or updates the parameters of the model being optimized based on the current iteration of the optimization process. These parameters could include weights, biases, learning rates, or other configurable settings that affect the model's behavior. Regarding the "marker" (a signal or trigger), each client application initiates the start of a scenario step, which represents a specific segment or phase of the overall optimization process. This step might involve executing a predefined set of actions, such as...This includes simulating a driving scenario, processing a data set, or performing a computational task. During the scenario step, the client application records relevant data that captures the current state of the system or environment. This data can include input features, model predictions, performance metrics, or any other information necessary for analyzing the model's behavior or performance. For example, in some implementations, a machine learning model for navigating autonomous vehicles in urban environments might be optimized. Each client application running on a specific client compute node updates the model's parameters based on the current iteration of the optimization algorithm. It then initiates the start of a scenario step, which simulates a specific driving scenario within an urban environment.During the scenario step, the client application records various data, such as the vehicle's position, speed, sensor readings, and environmental observations. This recorded data serves as input for evaluating the model's performance and for guiding further parameter updates in subsequent iterations of the optimization process.

[0081] Next, as part of step 4, each model task (e.g., model images 326, 336, and 346) receives such a tag, starts the scenario step, and records data. In the context of autonomous vehicle navigation, this step could involve simulating a segment of the route or performing a specific maneuver, such as turning or navigating an intersection. The scenario step represents a discrete unit of action within the overall scenario and can involve various operations, calculations, or interactions with the environment. This recorded data can include information such as the vehicle trajectory, sensor readings, environmental conditions, and any other factors that influence the vehicle's navigation.The model task then loads the recorded data, communicates the known data format to the corresponding client application, and marks the "Recording complete" flag. By specifying the data format, the model task provides the client application with essential information about how the recorded data can be effectively parsed and analyzed. This flag indicates that the recorded data is ready for further processing or analysis by the client application or other system components.

[0082] In step 5, each client compute node calculates one or more optimization KPIs and updates the one or more initial parameters (e.g., by training a model). In some implementations, each client compute node calculates a unique or different KPI. In alternative implementations, each client compute node calculates the same KPI. The local optimizer within each client application calculates various KPIs based on the data recorded during the scenario step. These KPIs serve as metrics for evaluating the performance of, for example, the autonomous navigation system. Optimization KPIs assess the relative improvement achieved by optimization algorithms or processes in improving the model's performance. They measure the effectiveness and efficiency of optimization efforts in the iterative refinement of the model.

[0083] Examples of different optimization KPIs include the following: Convergence rate, the rate at which the optimization algorithm converges to an optimal solution. It indicates how quickly the algorithm reaches a stable solution. Convergence accuracy indicates the degree of similarity between the optimized solution obtained by the algorithm and the true optimal solution. It measures the accuracy of the convergence achieved through the optimization process. Objective function value improvement indicates the reduction in the value of the objective function being optimized (e.g., loss function, cost function) across optimization iterations. It quantifies the improvement in model performance achieved through optimization. Error metric reduction indicates the reduction in error metrics (e.g., mean squared error, classification error rate) between the optimized model predictions and the ground-truth data.It indicates the reduction in prediction errors achieved through optimization. Convergence speed refers to the time the optimization algorithm needs to converge to an optimal solution. It measures the efficiency of the optimization process in achieving convergence within a specified timeframe. The KPI stability of the optimization process refers to the consistency and robustness of the optimization process across multiple iterations or variations. It assesses the reliability of the optimization algorithm in producing consistent results. Resource utilization efficiency refers to the degree of effective use of computing resources (e.g., CPU time, memory usage) during the optimization process. It evaluates the efficiency of resource allocation and utilization in optimization tasks.

[0084] Examples of KPIs in this context could include metrics such as: accuracy of virtual vehicle trajectory prediction, collision avoidance effectiveness, route planning and execution efficiency, and driving smoothness. After calculating the KPIs, the local optimizer analyzes these metrics to determine how effectively the model's current parameters are performing. If the KPIs indicate suboptimal performance or deviations from the desired goals (e.g., high collision rates, inaccurate trajectory predictions), the local optimizer adjusts the model parameters accordingly. In some implementations, parameter adjustments involve fine-tuning neural network weights, updating control policies, modifying decision algorithms, or making other changes to improve the autonomous navigation system's performance.

[0085] In an illustrative example, it is assumed that the KPIs calculated by the local optimizer show that the autonomous vehicle frequently deviates from its intended path during simulated driving scenarios. In response to these KPIs, the local optimizer can adjust the model's trajectory prediction parameters to improve its accuracy and reduce deviations. For example, it can increase the weight assigned to historical trajectory data, adjust the model's learning rate, or introduce regularization techniques to prevent overfitting.

[0086] In step 6, each of the client compute nodes 406 then transmits its own log data (i.e., the KPI and the updated parameters) back to the server 404. Each client compute node continues to perform local optimization concurrently (e.g., steps 3 through 6) until the threshold for the number of steps for local optimization is met and / or the optimization tolerance (e.g., a loss value via the loss function calculation) is satisfied. For example, after an initial epoch or round of training, each client compute node 406 transmits an updated parameter, and this occurs continuously after each epoch until a model is fully trained. In step 7, in response, the server 404 computes and transmits the "best" parameter to each of the client compute nodes 406, based on aggregating, in any appropriate way, each of the updated parameters received from each of the client compute nodes 406.This process involves the shared use of key performance indicators (KPIs) and model parameters between clients and the server to coordinate global optimization efforts. Such calculations and the transfer from step 7 constitute the "global optimization" as described herein (e.g., as performed by the global optimization component).

[0087] In an illustrative example of the calculations in Step 7, assume that a client application has calculated KPIs that indicate high accuracy in predicting the trajectory but low efficiency in avoiding collisions. It sends these KPI values, along with the corresponding model parameters, to Server 404. Server 404 coordinates the global optimization process by collecting KPIs and parameters from all client applications. It waits until all client applications have transmitted their data, ensuring that it has received information from every client compute node involved in the optimization task. Once Server 404 has collected KPIs and parameters from all client compute nodes, it aggregates this information to provide a comprehensive overview of the optimization process.It can analyze the collected data to identify trends, patterns, or areas that need improvement across different client nodes. Based on the aggregated results and optimization goals, the 404 server determines the best set of parameters to globally optimize the performance of the autonomous navigation system. It then sends these optimized parameters back to the respective client applications. For example, if the aggregated data indicates that a particular set of parameters consistently leads to improved performance across multiple nodes, the 404 server selects these parameters as the best choice.

[0088] In an illustrated example, assume that the 404 server receives KPIs and parameters from multiple client applications. After aggregating the data, it identifies that certain parameters consistently lead to better trajectory prediction and improved collision avoidance performance across all client 406 compute nodes. The 404 server then selects these optimized parameters and redistributes them to all client 406 compute nodes for further use in the optimization process. In step 8, each client 406 compute node takes this "best" parameter and integrates it in further rounds of local optimization (steps 3 through 6) in a loop until the total number of iterations is reached or the optimal tolerance is achieved (e.g., the loss over a loss function is below a threshold).

[0089] In step 9, the 404 server autogenerates one or more reports and transmits them back to the user device 402. For example, a large language model (LLM) can be used to generate a summary of the results, findings, and insights obtained from the optimization process. These reports can consolidate key information, metrics, visualizations, and analysis results into a structured document to facilitate review and presentation. For instance, after completing a series of optimization iterations or evaluation runs, the 404 server automatically generates a report summarizing the autonomous navigation system's performance. This report can include metrics such as trajectory prediction accuracy, collision rates, route planning efficiency, and traffic rule compliance.The report may also include details of the optimized parameters selected for the model based on the optimization process. It may contain graphs and visualizations of data and performance trends, such as line graphs showing changes in performance metrics over multiple iterations or bar charts comparing different parameter settings. Furthermore, the report may include insights derived from the analysis of the collected data, highlighting the strengths, weaknesses, and areas for improvement of the autonomous navigation system. Finally, the report may include recommendations—suggestions for further optimization steps, model parameter settings, or areas of focus for future development efforts.

[0090] Fig. Figure 5 is a sequence diagram illustrating how a user device, a server, and client compute nodes work together to perform local and global evaluation (e.g., benchmarking and / or validation), according to some embodiments. In some embodiments, the user device 502 represents the user device 302 of Fig. 3 and / or the user device 402 of Fig. 4. In some embodiments, server 404 represents the compute node 304 of the central server of Fig. 3 and / or the server 404 from Fig. 4. In some embodiments, the client compute nodes 320, 330 and 340 are of Fig. 3 in the client compute node 506 of Fig. 5 included. In some embodiments, the client compute nodes 506 represent the same set of client compute nodes 406 of Fig. 4. In other words: In some embodiments, Fig. 4 and Fig. 5 can be combined in such a way that the components can perform both local optimization and validation / benchmarking as well as global optimization and validation / benchmarking.

[0091] In step 1, the user device 402 transmits a scenario, a model type, and one or more initial parameters to the server 504. Regarding the model type, the user can, for example, provide their version of a known model, such as a specific radar model for DriveSim. Fig. 5 can use this model to calculate predefined KPIs for different existing scenarios in order to benchmark / validate the provided model per scenario and KPI, as described below.

[0092] In step 2, the server transmits the scenario, the model type (e.g., a model type ID that identifies the model type), and the one or more initial parameters to each of the client compute nodes 506. In steps 3 through 6, each of the client compute nodes 506 then performs local validation / benchmarking. In some implementations, steps 3 through 5 are identical to steps 3 through 5 of Fig. 4. That is, each client compute node first initializes the model task (step 3), then records data according to the user-specified scenario (step 4), and then calculates one or more KPIs in step 5 (except that the KPIs are evaluation KPIs). In some implementations, the KPIs in step 5 can be Fig. 5 may be suitable for benchmarking and / or validation and may therefore differ from the optimization KPIs as defined in relation to Fig. 4 are described. Benchmarking KPIs, for example, evaluate the absolute performance of a model against a standard or reference, often compared to existing models or ground-truth data. They provide insights into how well the model performs in absolute terms. Suitable benchmarking KPIs include, for example, accuracy – the percentage of correctly detected objects or obstacles in radar sensor data compared to ground truth; the error rate in estimating the position, speed, or trajectory of detected objects; precision – the proportion of true positive detections among all positive detections; and recall – the proportion of true positive detections among all actual positives (i.e.,Ground truth), false positive rate (the percentage of objects or obstacles falsely detected in radar sensor data relative to all detections), false negative rate (the percentage of actual objects or obstacles missed in radar sensor data relative to all actual objects), and the like. Other examples include a sensor's detection range (e.g., the maximum distance at which the radar sensor can reliably detect objects or obstacles), localization errors, velocity estimation errors, reaction time, or processing time. This differs from the optimization KPIs described in step 5 of [section / documentation]. Fig. 4 are illustrated.

[0093] Validation KPIs, on the other hand, assess a model's generalizability and reliability on unseen data to ensure that the model performs well in real-world scenarios beyond the training or benchmarking dataset. Examples of validation KPIs include: Accuracy (the percentage of correctly predicted results on the validation dataset, measuring the overall correctness of the model's predictions), Precision and Recall, F1 Score (the harmonic mean of Precision and Recall, providing a balance between these two, especially when dealing with unbalanced datasets), and Area Under the ROC Curve (AUC-ROC), a measure of the model's ability to distinguish between positive and negative classes across different thresholds.It assesses the overall classification performance of the model), mean squared error (MSE) (average of the squared differences between the values ​​predicted by the model and the actual values ​​in regression tasks. It quantifies the model's average prediction error), mean absolute error (MAE) (average of the absolute differences between the values ​​predicted by the model and the actual values ​​in regression tasks. It provides a measure of the average magnitude of errors in predictions), classification error rate (percentage of incorrectly classified instances in the validation dataset. It measures the proportion of incorrect predictions made by the model), confusion matrix (matrix that summarizes the performance of a classification model by comparing the predicted and actual class labels).It provides insights into the model's performance across different classes), cross-validation scores (average performance scores obtained through cross-validation techniques (e.g., k-fold cross-validation). It assesses the model's stability and generalizability across multiple validation subsets), bias and variance (analysis of bias (underfitting) and variance (overfitting) in the model's predictions on the validation dataset. It helps identify the trade-off between model complexity and generalizability performance).

[0094] In step 6, each of the client compute nodes 506 transmits its current KPIs (which may be the same or different) and updated parameters to the server 504. The purpose is to provide the server 504 with real-time updates regarding the performance of the models being evaluated or benchmarked in different scenarios or iterations. For example, suppose multiple client applications are running simulations with different radar models, such as in the context of simulating autonomous vehicles. During the simulations, each client continuously logs KPIs, such as detection accuracy, false positive rate, and localization error, along with the corresponding model parameters (e.g., neural network weights, configuration settings).This log data is transmitted to Server 504 in real time, enabling it to monitor the performance of each radar model in different scenarios and to make dynamically informed decisions.

[0095] In step 7, server 504 then aggregates the results from the individual client compute nodes 7, indicating global validation and / or benchmarking (e.g., as used by module 308 for global evaluation of Fig. 3. Therefore, Server 505 evaluates the aggregated data to perform global assessments or validations for all clients / client compute nodes 506. Server 504 waits to receive data from all client compute nodes 506 before proceeding with the aggregation and validation process. Once the data from all clients has been collected, the server aggregates the KPIs and parameters for all client compute nodes 506 to perform a comprehensive analysis. In some embodiments, for example, Server 504 calculates summary statistics, compares performance metrics for different models or scenarios, and identifies patterns or trends in the aggregated data. The server then uses this aggregated information to perform global validation assessments, such as comparing the performance of different radar models or identifying common problems across simulations.

[0096] In another example of the global evaluation from Step 7, consider an online marketplace that uses a distributed system to collect and analyze user behavior data from multiple client applications (e.g., mobile apps, web browsers). The marketplace operator wants to perform global validation of user activity across different regions. Each of the client 506 compute nodes collects user activity metrics, such as page views, click-through rates, and conversion rates, from users in a specific geographic region. These client applications continuously send the collected metrics to the server 504. The server 504 acts as a global validator, responsible for aggregating and analyzing the user activity data from all regions. The server 504 waits until it has received data from all client applications / client 506 compute nodes across different regions.Once data from all customers has been collected, the 504 server aggregates user activity metrics, including page views, click-through rates, and conversion rates, from each region. The 504 server calculates summary statistics (e.g., average page views per user, overall click-through rate) and identifies trends or patterns in user behavior across different regions. Using this aggregated data, the 504 server performs a global validation of user activity metrics across the entire online marketplace. It compares user activity levels between regions, identifies regions with higher or lower activity rates, and assesses the overall performance of the online marketplace.Based on the results of the global validation, the marketplace operator can make informed decisions to optimize the user experience, implement targeted marketing measures, or allocate resources effectively. For example, if the global validation reveals that certain regions have lower conversion rates, the operator can focus on improving user engagement strategies or offering targeted promotions in those regions.

[0097] In step 8, the server 504 autogenerates one or more reports and sends them to the user device 502. For example, once data is received from all client compute nodes 506, the server 504 automatically generates comprehensive reports summarizing the performance evaluation results for each model. These reports include detailed analyses of the models' performance metrics, comparisons of different model configurations or algorithms, and insights into each model's effectiveness in making predictions. In some implementations, visualizations such as charts, graphs, and tables are included to effectively present the performance evaluation findings. The reports may also include recommendations for further model refinement or optimization based on the observed performance trends and analyses.

[0098] Fig. Figure 6 is a flowchart of an exemplary method 600 for optimizing a parameterizable model, according to some embodiments. In some embodiments, the method 600 is executed by the compute node 304 of the central server and / or a client compute node (e.g., 320 of Fig. 3) of Fig. 3. Each block of the procedure 600 (and / or 700) described herein comprises a computational process that can be performed using any combination of hardware, firmware, and / or software. Various functions can be performed, for example, by a processor executing instructions stored in memory. The procedures can also be embodied as computer-usable instructions stored on computer storage media. The procedures can be provided by a standalone application, a service or hosted service (standalone or in combination with another hosted service), or a plug-in for another product, to name just a few. Furthermore, procedure 600 is exemplified with respect to the client compute nodes 406 of Fig. 4 described. However, these procedures can additionally or alternatively be performed by any system or any combination of systems, including, but not limited to, the systems described herein.

[0099] In some embodiments, block 602 receives user input containing at least one of the following: a scenario, one or more initial parameters, or configuration data associated with the optimization of a parameterizable model. For example, the scenario might contain a user instruction defining a sequence of actions associated with the optimization, where at least one action in the sequence is a capture or recording task associated with the optimization of the parameterizable model. Examples of such user instructions are provided by the "scenario" as part of the input in step 1 of Fig. 4 described. Further examples include the scenario in user input 104 of Fig. 1. In an illustrative example of such a scenario, engineers in the aerospace industry frequently conduct flight simulations to evaluate the performance of aircraft under various conditions. These simulations involve a series of predefined steps or actions that represent different flight scenarios. Each scenario is designed to test specific aspects of the aircraft's performance, such as maneuverability, stability, and fuel efficiency. A scenario, in this case, can be defined as a sequence of flight maneuvers and conditions that simulate real-world flight scenarios. Each scenario includes parameters such as altitude, airspeed, throttle setting, and control inputs. For example, a scenario might simulate a sequence of actions including takeoff, climb to cruising altitude, turns, descent, and landing.During flight simulation, data logging steps are performed to determine various flight parameters and performance metrics. These steps can include recording data such as altitude, airspeed, engine power, control surface positions, and environmental conditions (e.g., wind speed, temperature).

[0100] In some embodiments, such “configuration data” included in the user input can contain any suitable data as described herein, such as a number of compute nodes to use for optimization (e.g., the user specifying to limit the compute nodes for optimization to 5), a specification of whether to perform optimization or validation (e.g., optimization in this context), a current parameter to be evaluated by the parameterized model, and a number of steps associated with the optimization. In some embodiments, such a set of steps refers to or specifies the number of iterations or epochs the optimization algorithm will go through. For example, it can control the number of training epochs for each hyperparameter combination during grid search. Other examples of configuration data include the learning rate (e.g.,[0,001, 0,01, 0,1]), Batch size: (e.g. [32, 64, 128]), Dropout rate: (e.g. [0,2, 0,4, 0,6]) or the like.

[0101] In Block 604, in response to receiving user input, data is recorded or captured in some embodiments according to the user input (e.g., the scenario). For example, using the illustration above, in some embodiments data such as altitude, airspeed, engine power, control surface positions, and environmental conditions (e.g., wind speed, temperature) are recorded during takeoff, climb, and cruising altitude of a simulated airspace model. In some embodiments, Block 604 represents step 4 of Fig. 4. In some embodiments, block 604 alternatively represents an “instruction” to perform such data recording or acquisition, or otherwise a signal containing one or more initial parameters or the scenario as described in step 2 of Fig. Figure 4 illustrates this. In other words, block 604 can represent step 2, which in some embodiments is performed by server 404.

[0102] In some embodiments, block 606 updates a parameter during the optimization of the parameterizable model, based on at least the recording or acquisition of at least some of the data (in block 604). In some embodiments, block 606 represents the local and / or global optimization as described herein. Examples of block 606 are given in relation to the "Updating One or More Initial Parameters" section of step 5 in Fig. 4. For example, during the climb phase, the aircraft's speed parameters must be adjusted to maintain a specified rate of climb while simultaneously reaching the desired altitude, as illustrated in the flight simulation example above. The flight simulation model interacts with the simulation environment to dynamically adjust the aircraft's speed parameters based on predefined criteria and inputs. For instance, the simulation model can incorporate feedback from control inputs, environmental conditions (such as air density and wind speed), and aircraft performance characteristics to determine the optimal speed for the climb. During the climb phase of the flight scenario, the simulation model updates the aircraft's speed parameters to achieve the desired rate of climb and altitude.This can include increasing or decreasing the thrust lever setting, adjusting the pitch angle, and managing other control surfaces to control the aircraft's speed.

[0103] In some embodiments, process 600 additionally includes the calculation of an optimization performance metric (e.g., an optimization KPI, as in relation to step 5 of Fig. 4 described). And at least partially based on the computation of the performance metric and the updating of the parameter during the optimization, in some embodiments the parameter and the optimization performance metric are then transferred as part of the optimization to a first compute node of a plurality of compute nodes. Examples of this are given with respect to steps 5 and 6 of Fig. As described in section 4, a client compute node in step 406 computes an optimization KPI and updates the one or more updated initial parameters for step 6, and then transmits the log data in step 6 to server 404. In an illustrative example, a client compute node might transmit the calculated convergence accuracy value ε=0.001 along with a value for an updated parameter. In some embodiments, method 600 alternatively or additionally includes "receiving" such an optimization performance metric and an updated parameter instead of merely "transmitting" such data.

[0104] In response to such a transmission of the updated parameter to the first compute node of the plurality of compute nodes as part of Method 600, in some embodiments a set of values ​​is received from the first compute node. In some embodiments, the parameter can then be modified according to the set of values, based at least on the first compute node with aggregated parameter values ​​from the plurality of compute nodes. An example of such functionality is given with respect to step 8 of Fig. Section 4 describes how a client compute node in 406 updates a parameter based on having received the "best" parameter from server 404 via step 7. In an illustrative example of the first node with aggregated parameter values, server 404 may have received a learning rate from each client compute node, and in response, server 404 calculates an average of each learning rate received from client compute node 406. Server 404 then transmits such an average to all client compute nodes 406, which is a new starting point (e.g., training epoch) for all parameters.

[0105] In some embodiments, Method 600 includes recording or capturing at least a portion of the data for at least one second action in the sequence of actions, according to user instructions, based at least on the parameter update during the optimization task. In other words, the parameter update influences an action. Suppose, for example, that this is a neural network model trained using a stochastic gradient descent (SGD) optimization algorithm. In each iteration of the training process, various embodiments update the model's parameters based on a small batch of training data. Initially, embodiments begin by initializing the neural network model with random weights and biases.During each iteration of the training loop, in various embodiments: (1) a mini-batch of training examples is randomly selected from the dataset; (2) the gradients of the loss function with respect to the model parameters are calculated using the selected mini-batch; (3) the model parameters (weights and bias) (e.g., step 6 of . Fig. 4) updated using the calculated gradients and the learning rate according to the SGD algorithm. As part of the training process, some embodiments record various data relating to the model's performance and behavior. For example, in some embodiments, optimization metrics such as training loss, accuracy, and validation loss are recorded after each training iteration. This data is recorded using the updated model parameters, which are set based on the mini-batch of training data processed in that iteration.

[0106] In an illustrative example, during a training iteration, some embodiments update the weights and biases of the neural network model based on a mini-batch of 32 training examples. After updating the parameters, some embodiments calculate the training loss and accuracy using the updated model. The recorded data for this iteration, such as training loss and accuracy, are mapped to the updated model parameters used in the calculation. This ensures that the recorded data accurately reflects the model's performance with the latest parameter updates. In this example, the data recorded in each training step are mapped to the updated optimizer parameters, representing the current state of the model after processing a mini-batch of training data.This ensures that the recorded data accurately reflects the model's performance and behavior throughout the entire training process.

[0107] In some embodiments, method 600 includes initiating the presentation of data (e.g., the autogenerated report, as in relation to step 9 of Fig. 4 described). For example, in some embodiments (e.g., by the client compute nodes 406 and / or the server 404), at least one of one or more parameter values, one or more performance metrics for optimization is presented to the data or user instruction at a user device.

[0108] Fig. Figure 7 is a flowchart of an exemplary method 700 for validating and / or benchmarking a parameterizable model, according to some embodiments. In some embodiments, the method 700 is executed by the compute node 304 of the central server and / or a client compute node (e.g., 320 of Fig. 3) performed. In Block 703, in some embodiments, user input is received that contains at least one of a scenario or a type of parameterizable model. For example, in some embodiments, a string specifying a particular type of machine learning model is received via a CLI. Examples include linear regression, where the parameters contain coefficients for each feature; logistic regression, where the parameters contain coefficients for each feature and an intercept; support vector machines (SVMs), where the parameters contain the choice of kernel function and the regularization parameter; decision trees, where the parameters contain the tree depth, the minimum sample size per leaf, and the splitting criterion; and random forests, where the parameters contain the number of trees, the maximum tree depth, and the number of features considered for each split.Neural networks, where the parameters include the architecture (number of layers, number of neurons per layer), activation functions, learning rate, and regularization strength.

[0109] In another example, the user can alternatively or additionally specify a type of simulation model. For example, physics-based models, where the parameters include physical constants, material properties, initial conditions, and boundary conditions. In another example, the user-specified type of parameterizable model can include control systems, such as PID controllers, where the parameters include proportional, integral, and derivative gains. In yet another example, the user-specified models can include optimization models, such as linear programming models, where the parameters include objective function coefficients and constraints. Further examples include image processing models (e.g., object recognition models, where the parameters include anchor box sizes, aspect ratios, and confidence thresholds). Other examples include natural language processing models (e.g.,...large language models (LLMs), Long Short Term Memory (LSTM) models), where the parameters include the number of layers, the attention heads, and the dropout rates.

[0110] In some embodiments, Block 705 calculates one or more evaluation performance metrics, at least partially based on user input. Examples of such calculations include the calculation of the evaluation KPIs in step 5 of Fig. 5. For example, in some embodiments, a dictionary, lookup table, hash table, or other key-value pair data structure is used to map the specific user-specified type of model to a corresponding key in the data structure in order to derive corresponding values ​​that specify the specific evaluation KPIs used for that model. In an illustrative example, the user may specify that they are validating an "LLM." Accordingly, in certain embodiments, using this string, such a string is matched against a corresponding "LLM" key string in a lookup table. In various embodiments, the values ​​in the same entry are then read. In some embodiments, such values ​​are, for example, one or more of the following: Perplexity (measures how well the model predicts a sequence of words).Lower perplexity indicates better language understanding), Word Error Rate (WER) (measures the percentage of incorrectly predicted words compared to the ground truth), Semantic Proximity (measures the proximity between predicted and reference sentences using semantic embedding or proximity metrics), Gap Test Accuracy (measures how accurately the model predicts missing words in a given context), Named Entity Recognition (NER) Accuracy (measures the accuracy of identifying named entities in the text), Sentiment Analysis Accuracy (measures the accuracy of predicting sentiment labels (positive, negative, neutral) for given text samples), Bias Metrics (measures the presence of bias in the model's predictions, such as...Gender bias, ethnic bias, or socioeconomic bias), inference time (measures the time required to generate responses or predictions for given inputs). Accordingly, in response to the reading of one or more of these values, in some implementations the corresponding validation KPIs are performed on a parameterizable model output (e.g., a final prediction of the model after the model has been trained).

[0111] In Block 707, based on the calculation of at least one or more evaluation performance metrics, the parameterizable model is validated and / or benchmarked in some embodiments. In some embodiments, Block 707 represents local and / or global validation and / or benchmarking, as described herein. Consider a situation in which a text generation model, such as a large language model (LLM), has been generated and its performance needs to be validated using various KPIs. In different embodiments, specific evaluation tasks are first defined to assess different aspects of the text generation model, such as fluency, coherence, and relevance. The tasks may include generating text completions for given prompts, generating responses to conversational prompts, or generating summaries for given documents.

[0112] Based on the defined evaluation tasks, relevant KPIs are selected in various embodiments to measure the performance of the model (e.g., as described above in relation to the data structure of the key-value pairs).

[0113] For example, in various implementations, perplexity, coherence scores, and human judgment scores can be chosen as KPIs to assess language comprehension and generation quality.

[0114] Continuing this LLM example, a dataset is prepared containing evaluation examples for each evaluation task, along with reference outputs or ground truth for comparison. The dataset covers various language patterns, topics, and styles to provide a comprehensive evaluation of the model in some embodiments. In various embodiments, the evaluation data is then fed into the text generation model, and outputs are generated for each evaluation example. For each generated output, selected KPIs are calculated in various embodiments to quantify the model's performance. For perplexity, in some embodiments, the perplexity numerical value for each generated sequence is calculated using the model's language model probabilities and compared to the ground truth. For coherence numerical values, automated metrics, such as...Coherence or readability scores, used to quantify the logical flow and coherence of the generated text.

[0115] In some embodiments, each individual client compute node aggregates the KPI numerical values ​​obtained by evaluating multiple examples in different evaluation tasks. This provides an overall assessment of the model's performance across various dimensions. In other embodiments, the KPI numerical values ​​are interpreted to understand which aspects of the model's performance can be improved and which aspects are satisfactory. In other embodiments, the performance of the text generation model is compared to base models or existing prior art models using the same evaluation tasks and KPIs. This helps to benchmark the model's performance against existing approaches.Based on the analysis and interpretation of the results, in some embodiments the text generation model is iteratively refined to improve its performance in areas where weaknesses have been identified. This may involve adjusting the model architecture, hyperparameters, or training procedures based on the insights gained from the evaluation. By following these steps, a text generation model can be systematically validated and benchmarked in various embodiments based on the execution of one or more KPIs, providing insights into its strengths, weaknesses, and areas for improvement.

[0116] In some embodiments, process 700 includes recording or capturing data according to at least one action of a sequence of actions specified in the user input scenario of block 703. For example, as with respect to Fig. As described in section 5, a client compute node performs step 4. Based at least partially on the calculation of one or more evaluation performance metrics and the recording or capture of the data according to the at least one action, in some embodiments the one or more evaluation performance metrics are transferred to a first compute node of a plurality of compute nodes. In other words, in response to at least one validation or benchmarking action of the parameterizable model (in block 707), in some embodiments the one or more evaluation performance metrics are transferred to a first compute node of a plurality of compute nodes. Examples of this are described with respect to one of the client compute nodes 506, which performs step 6 to transfer the current KPI and the updated parameter to the server 504.Although this step is described in relation to the client compute nodes 506, in additional or alternative embodiments the server 504 receives (instead of "transmits") such a current KPI and / or an updated parameter.

[0117] In response to the transmission (and / or reception) of one or more evaluation performance metrics to / from the first compute node of the plurality of compute nodes, in some embodiments (e.g., a client compute node), the first compute node (e.g., Server 504) receives (and / or generates) an indication (e.g., a flag) of whether global validation has occurred, based at least on the first compute node with aggregated performance metrics from the plurality of compute nodes. For example, Server 504 may have different or unique performance metrics and / or a different scenario from each of the Client Compute Nodes 506. Server 504 may, for example, receive different / unique KPIS from each Client Compute Node (e.g.,The server receives a 504 error: a first KPI and / or a first scenario from a first client compute node and a second KPI and / or a second scenario from a second client compute node.

[0118] Server 504 can then aggregate these evaluation KPIs and their corresponding values. For example, after completing the site validation tasks, each client compute node reports its validation results, including the calculated evaluation performance metrics, back to Server 504. Server 504 waits to receive the validation results from all distributed client compute nodes 506 before aggregating the results. Once all client compute nodes 506 have submitted their validation results, Server 504 aggregates the results to generate a comprehensive validation report. For example, Server 504 can aggregate the results by calculating the average accuracy, average precision, average recall, and other metrics for all client compute nodes 506 to provide a consolidated overview of model performance.In some embodiments, aggregating the results alternatively or additionally includes consolidating or bundling each individual assessment KPI from each of the client compute nodes into a report without performing any further calculations.

[0119] Based on the aggregated validation results, Server 504 generates a global validation report summarizing the overall performance of the model across all distributed client compute nodes 506. The report can include statistics, visualizations, and insights derived from the aggregated validation metrics.

[0120] Stakeholders can use such a global validation report to assess the model's performance and make informed decisions about further improvements or adjustments. Feedback from the validation report can lead to iterative refinement of the model or the validation process to address identified issues or weaknesses. This aggregation process ensures that the global validation report reflects collective validation efforts across all distributed client compute nodes and provides a unified perspective on model performance.

[0121] Fig. Figure 8 illustrates a visualization 800 (e.g., a user interface) of KPI values ​​and parameter values ​​across optimization iterations, according to some embodiments. In some embodiments, the visualization 800 represents at least part of what is shown in the autogenerated reports in step 9 of Fig. 4 and / or step 8 of Fig. 5 is produced.

[0122] Visualization 800 contains a first linear graph, 800, where the Y-axis represents KPI values ​​(e.g., optimization performance metrics or evaluation performance metrics) and the X-axis represents optimization iterations or parameter values. An "optimization iteration," as described herein, can refer to any suitable iteration, such as a training epoch / round (e.g., "10" refers to 10 epochs), a sequence of timestamps or time periods (e.g., "10" refers to 10 minutes), and / or the like. Visualization 800 also contains a second set of linear graphs, 810, 812, and 814, which represent individual trends (e.g., parameters). A trend is a collection of data points, each of which represents a specific configuration of parameter values, their associated KPI values, and / or optimization iterations.Each data point in the trend graph corresponds to a separate trial, parameter, iteration, or evaluation of the parameterizable model. A trend graph represents the evolution or trajectory of the model's performance (KPI values) as the parameters are adjusted or optimized over time. During model optimization or validation processes, trends are recorded at each iteration or trial in various implementations by logging parameter values ​​and their corresponding KPI values. Trends can be recorded for various purposes, such as monitoring model convergence, analyzing parameter sensitivity, or comparing different optimization algorithms. The Y-axis (820) represents the varying parameter values ​​of three different parameters. The X-axis (822) represents different optimization iterations. EXEMPLARY AUTONOMOUS VEHICLE

[0123] Fig. Figure 9A is an illustration of an exemplary autonomous vehicle 900, according to some embodiments of the present disclosure. The autonomous vehicle 900 (here alternatively referred to as "vehicle 900") may, without limitation, include: a passenger vehicle, such as a car, truck, bus, emergency service vehicle, shuttle, electric or motorized bicycle, motorcycle, fire engine, police vehicle, ambulance, boat, construction vehicle, underwater vehicle, robotic vehicle, drone, aircraft, a vehicle coupled to a trailer (e.g., a semi-trailer truck used for transporting cargo), and / or another type of vehicle (e.g., one that is unmanned and / or carries one or more passengers).Autonomous vehicles are generally described in terms of automation levels defined by the National Highway Traffic Safety Administration (NHTSA), a division of the U.S. Department of Transportation, and the Society of Automotive Engineers (SAE) standard "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles" (Standard No. J3016-201806, published June 15, 2018, Standard No. J3016-201609, published September 30, 2016, and earlier and future versions of this standard). The Vehicle 900 can exhibit functionality corresponding to one or more of the Levels 3 through 5 of autonomous driving levels.The Vehicle 900 can exhibit functionality corresponding to one or more of the Levels 1 to 5 of autonomous driving. For example, depending on its configuration, the Vehicle 900 may be capable of driver assistance (Level 1), partial automation (Level 2), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5). The term "autonomous," as used here, may encompass any and / or all types of autonomy for the Vehicle 900 or any other machine, such as fully autonomous, highly autonomous, conditionally autonomous, partially autonomous, assistive autonomy, semi-autonomous, primary autonomous, or any other designation.

[0124] The vehicle 900 can include components such as a chassis, a vehicle body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other vehicle components. The vehicle 900 can include a propulsion system 950, such as an internal combustion engine, a hybrid electric power plant, a pure electric motor, and / or another type of propulsion. The propulsion system 950 can be connected to a drivetrain of the vehicle 900, which may include a transmission to enable the propulsion of the vehicle 900. The propulsion system 950 can be controlled in response to signals received from the throttle valve or accelerator device 952.

[0125] A steering system 954, which may include a steering wheel, can be used to steer the vehicle 900 (e.g., along a desired path or route) when the drive system 950 is in operation (e.g., when the vehicle is in motion). The steering system 954 can receive signals from a steering actuator 956. The steering wheel is optional for full automation (level 5).

[0126] The brake sensor system 946 can be used to actuate the vehicle brakes in response to receiving signals from the brake actuators 948 and / or the brake sensors.

[0127] The one or more controllers 936, the one or more systems-on-chips (SoCs) 904 ( Fig. 9C) and / or GPUs, can supply signals (e.g., representing instructions) to one or more components and / or systems of the vehicle 900. For example, the one or more controllers can send signals to actuate the vehicle brakes via one or more brake actuators 948, to actuate the steering system 954 via one or more steering actuators 956, and to actuate the propulsion system 950 via one or more throttle / accelerator devices 952. The one or more controllers 936 can include one or more built-in (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and issue operating commands (e.g., signals representing commands) to enable autonomous driving and / or to assist a human driver in driving the vehicle 900.The one or more Controller 936s can include a first Controller 936 for autonomous driving functions, a second Controller 936 for functional safety functions, a third Controller 936 for artificial intelligence functions (e.g., computer vision), a fourth Controller 936 for infotainment functions, a fifth Controller 936 for emergency redundancy, and / or other controllers. In some examples, a single Controller 936 can perform two or more of the above-mentioned functionalities, two or more Controller 936s can perform a single functionality, and / or any combination thereof.

[0128] The one or more controllers 936 can provide the signals for controlling one or more components and / or systems of the vehicle 900 in response to sensor data received from one or more sensors (e.g. sensor inputs). The sensor data can be received, for example, without restriction, by one or more of the following: Global Navigation Satellite Systems (GNSS) sensor(s) 958 (e.g., Global Positioning System sensor(s)), radar sensor(s) 960, ultrasonic sensor(s) 962, LiDAR sensor(s) 964, inertial measurement unit (IMU) sensor(s) 966 (e.g., accelerometer(s), gyroscope(s), magnetic compass(s), magnetometer(s), etc.), microphone(s) 996, stereo camera(s) 968, wide-angle camera(s) 970 (e.g., fisheye cameras), infrared camera(s) 972, ambient camera(s) 974 (e.g.,360-degree cameras), long-range and / or medium-range camera(s) 998, speed sensor(s) 944 (e.g. for measuring the speed of the vehicle 900), vibration sensor(s) 942, steering sensor(s) 940, brake sensor(s) (e.g. as part of the brake sensor system 946), one or more occupant monitoring system (OMS) sensors 901 (e.g. one or more interior cameras) and / or other sensor types.

[0129] One or more of the controllers 936 can receive inputs (e.g., in the form of input data) from an instrument cluster 932 of the vehicle 900 and provide outputs (e.g., in the form of output data, display data, etc.) via a human-machine interface (HMI) display 934, an acoustic alarm, a loudspeaker, and / or via other components of the vehicle 900. The outputs can include information such as vehicle speed, engine speed, time, map data (e.g., the high-definition (HD) map 922 from Fig. 9C), location data (e.g., the location of vehicle 900, e.g., on a map), direction, location of other vehicles (e.g., an occupancy grid), information about objects and the status of objects as perceived by the one or more controllers 936, etc. For example, the HMI display 934 can show information about the presence of one or more objects (e.g., a road sign, a warning sign, a changing traffic light, etc.) and / or information about driving maneuvers that the vehicle has performed, is currently performing, or will perform (e.g., changing lanes now, taking exit 34B in two miles, etc.).

[0130] The vehicle 900 also includes a network interface 924, which can use one or more wireless antennas 926 and / or modems for communication over one or more networks. The network interface 924 can be suitable, for example, for communication via Long-Term Evolution (LTE), Wideband Code Division Multiple Access (WCDMA), Universal Mobile Telecommunications System (UMTS), Global System for Mobile Communication (GSM), IMT-CDMA Multi-Carrier (CDMA2000), etc. The one or more wireless antennas 926 can also allow communication between objects in the environment (e.g., vehicles, mobile devices, etc.) using local area networks such as Bluetooth, Bluetooth Low Energy (LE), Z-Wave, ZigBee, etc.and / or low power wide area networks (LPWANs), such as LoRaWAN, SigFox, etc.

[0131] Fig. 9B is an example of camera locations and fields of view for the exemplary autonomous vehicle 900. Fig. 9A, according to some embodiments of the present disclosure; The cameras and respective fields of view are an exemplary embodiment and are not intended to be limiting. For example, additional and / or alternative cameras may be included and / or the cameras may be located at different locations on the vehicle 900.

[0132] The camera types may include, but are not limited to, digital cameras designed for use with the components and / or systems of the Vehicle 900. The one or more cameras may operate at Automotive Safety Integrity Level (ASIL) B and / or another ASIL. Depending on the configuration, the camera types may be capable of any frame rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc. The cameras may use roller shutters, global shutters, another type of shutter, or a combination thereof.In some examples, the color filter array may include a red-clear-clear-clear (RCCC) color filter array, a red-clear-clear-blue (RCCB) color filter array, a red-blue-green (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor color filter array (RGGB), a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, cameras with clear pixels, such as cameras with an RCCC, RCCB, and / or RBGC color filter array, may be used to increase light sensitivity.

[0133] In some examples, one or more of the cameras can be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For instance, a multi-function monocular camera can be installed to provide features including lane departure warning, traffic sign recognition, and intelligent headlight control. One or more of the cameras (e.g., all cameras) can simultaneously record and provide image data (e.g., video).

[0134] One or more cameras can be mounted in a bracket, such as a specially designed (three-dimensional ("3D") printed bracket, to eliminate stray light and reflections from inside the vehicle (e.g., reflections of the dashboard in the windshield) that could interfere with the camera's image acquisition. Regarding the mounting of exterior mirrors, the mirrors can be individually 3D printed so that the camera mounting plate is shaped to fit the mirror. In some cases, the one or more cameras can be integrated into the exterior mirror. For side cameras, the one or more cameras can also be integrated into the four pillars at each corner of the cabin.

[0135] Cameras with a field of view that includes portions of the environment in front of the vehicle (e.g., forward-facing cameras) can be used for surround view to help identify forward paths and obstacles and to provide, with the aid of one or more Controller 936 and / or Control SoCs, information critical for creating an occupancy grid and / or determining preferred vehicle paths. Forward-facing cameras can be used to perform many of the same ADAS functions as LiDAR, including emergency braking, pedestrian detection, and collision avoidance. Forward-facing cameras can also be used for ADAS functions and systems that include lane departure warnings (LDW), autonomous cruise control (ACC), and / or other functions such as traffic sign recognition.

[0136] A variety of cameras can be used in a forward-facing configuration, including, for example, a monocular camera platform containing a complementary metal oxide semiconductor (CMOS) color imager. Another example is the 970 wide-angle camera series, which can be used to capture objects moving into view from the periphery (e.g., pedestrians, crossing vehicles, or bicycles). Although in Fig. While only one wide-angle camera is illustrated in Figure 9B, the vehicle 900 can contain any number (including zero) of wide-angle cameras 970. Furthermore, any number of long-range cameras 998 (e.g., a pair of long-range stereo cameras) can be used for depth-based object detection, particularly for objects for which a neural network has not yet been trained. The one or more long-range cameras 998 can also be used for object detection and classification, as well as basic object tracking.

[0137] Any number of stereo cameras 968 can also be included in a forward-facing configuration. In at least one embodiment, one or more of the stereo cameras 968 can include an integrated control unit comprising a scalable processing unit that can provide programmable logic (“FPGA”) and a multicore microprocessor with an integrated controller area network (“CAN”) or Ethernet interface on a single chip. Such a unit can be used to create a 3D map of the vehicle's surroundings that includes a distance estimate for all points in the image. Alternatively, one or more stereo cameras 968 can include a compact stereo vision sensor that may contain two camera lenses (one left and one right) and an image processing chip that measures the distance between the vehicle and the target object and processes the generated information (e.g.,Metadata) can be used to activate the autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 968 can be used in addition to or as an alternative to those described here.

[0138] Cameras with a field of view that includes sections of the environment to the sides of the vehicle 900 (e.g., side cameras) can be used for the surround view and provide information used to create and update the occupancy grid and to generate side-impact collision warnings. For example, one or more surround cameras 974 (e.g., four surround cameras 974, as in Fig. (9B illustrated) are positioned on the vehicle 900. The one or more surround-view cameras 974 can include one or more wide-angle cameras 970, one or more fisheye cameras, one or more 360-degree cameras, and / or the like. For example, four fisheye cameras can be mounted at the front, rear, and sides of the vehicle. In an alternative arrangement, the vehicle can use three surround-view cameras 974 (e.g., left, right, and rear) and one or more other cameras (e.g., a forward-facing camera) as a fourth surround-view camera.

[0139] Cameras with a field of view that includes sections of the area behind the vehicle 900 (e.g., reversing cameras) can be used for parking assistance, surround view, rear-impact warnings, and creating and updating the occupancy grid. A variety of cameras can be used, including cameras that are also suitable as one or more forward-facing cameras (e.g., one or more long-range and / or medium-range cameras 998, one or more stereo cameras 968, one or more infrared cameras 972, etc.), as described herein.

[0140] Cameras with a field of view that includes portions of the interior environment within the cabin of the vehicle 900 (e.g., one or more OMS sensors 901) can be used as part of an occupant monitoring system (OMS), such as a driver monitoring system (DMS), but are not limited to this use. For example, OMS sensors (e.g., the one or more OMS sensors 901) can be used (e.g., by the one or more controllers 936) to track the gaze direction, head position, and / or blinking of an occupant and / or driver. This gaze information can be used to determine the level of attention of the occupant or driver (e.g., to detect drowsiness, fatigue, and / or distraction) and / or to take appropriate action to prevent harm to the occupant or driver.In some embodiments, data from OMS sensors can be used to enable gaze-controlled operations triggered by the driver and / or other occupants, such as, but not limited to, adjusting cabin temperature and / or airflow, opening and closing windows, controlling cabin lighting, controlling entertainment systems, adjusting mirrors, adjusting seat positions, and / or other operations. In some embodiments, an OMS can be used for applications such as determining whether objects and / or occupants have been left in a vehicle cabin (e.g., by detecting the presence of occupants after the driver has left the vehicle).

[0141] Fig. 9C a block diagram of an exemplary system architecture for the exemplary autonomous vehicle 900 from Fig. 9A, according to some embodiments of the present disclosure. It should be noted that these and other arrangements described herein are presented only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, arrays, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as single or distributed components, or in conjunction with other components, in any suitable combination and at any suitable location. Various functions described herein, which are performed by entities, may be executed by hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory.

[0142] Each of the components, features and systems of the 900 vehicle in Fig. 9C is illustrated as being connected via bus 902. Bus 902 may contain a Controller Area Network (CAN) data interface (here alternatively referred to as a "CAN bus"). A CAN can be a network within the vehicle 900 that serves to support the control of various features and functions of the vehicle 900, such as the operation of brakes, acceleration, braking, steering, windshield wipers, etc. A CAN bus may be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., a CAN ID). The CAN bus can be read to determine the steering wheel angle, vehicle speed, engine speed (rpm), button positions, and / or other vehicle status indicators. The CAN bus may be ASIL B compliant.

[0143] Although the 902 bus is described here as a CAN bus, this is not intended as a limitation. For example, FlexRay and / or Ethernet can be used in addition to or as an alternative to the CAN bus. Furthermore, while a single line is used to represent the 902 bus, this is not meant as a restriction. There can be any number of 902 buses, which may contain one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using a different protocol. In some examples, two or more 902 buses can be used to perform different functions and / or for redundancy. For example, a first 902 bus can be used for collision avoidance functionality, and a second 902 bus can be used for actuation control.In each example, each bus 902 can communicate with one of the vehicle 900 components, and two or more buses 902 can communicate with the same components. In some examples, each SoC 904, each controller 936, and / or each computer within the vehicle can have access to the same input data (e.g., inputs from vehicle 900 sensors) and be connected to a common bus, such as the CAN bus.

[0144] The vehicle 900 can contain one or more controllers 936, as shown here in relation to Fig. 9A are described. The one or more 936 controllers can be used for a variety of functions. The one or more 936 controllers can be coupled with one or more of the various other components and systems of the 900 vehicle and can be used for controlling the 900 vehicle, for the artificial intelligence of the 900 vehicle, for infotainment for the 900 vehicle, and / or the like.

[0145] The vehicle 900 can contain one or more systems-on-a-chip (SoC) 904. The SoC 904 can contain one or more CPUs 906, one or more GPUs 908, one or more processors 910, one or more caches 912, one or more accelerators 914, one or more data storage devices 916, and / or other components and features not illustrated. The one or more SoCs 904 can be used to control the vehicle 900 in a variety of platforms and systems. For example, the one or more SoCs 904 in a system (e.g., the system of the vehicle 900) can be combined with an HD card 922, which is accessed via a network interface 924 by one or more servers (e.g., the one or more servers 978). Fig. 9D) may receive map refreshes and / or updates.

[0146] The one or more CPUs 906 can contain a CPU cluster or CPU complex (hereafter referred to as "CCPLEX"). The one or more CPUs 906 can contain multiple cores and / or L2 caches. In some embodiments, the one or more CPUs 906 can, for example, contain eight cores in a coherent multiprocessor configuration. In some embodiments, the one or more CPUs 906 can contain four dual-core clusters, each cluster having a dedicated L2 cache (e.g., a 2 MB L2 cache). The one or more CPUs 906 (e.g., the CCPLEX) can be configured to support the concurrent operation of clusters, allowing any combination of clusters of the one or more CPUs 906 to be active at any given time.

[0147] The one or more CPUs 906 can implement power management features that include one or more of the following: individual hardware blocks can be automatically clocked when idle to dynamically conserve power; each core clock can be controlled when the core is not actively executing instructions due to the execution of WFI / WFE instructions; each core can be independently power-controlled; each core cluster can be independently clock-controlled when all cores are clock-controlled or power-controlled; and / or each core cluster can be independently power-controlled when all cores are power-controlled. The one or more CPUs 906 can also implement an enhanced power state management algorithm that establishes acceptable power states and expected wake-up times, and the hardware / microcode determines the best power state to input for the core, cluster, and CCPLEX.The processing kernels can support simplified sequences for inputting the energy state into the software, thereby offloading the work to the microcode.

[0148] The one or more GPUs 908 can include an integrated GPU (referred to herein alternatively as an "iGPU"). The one or more GPUs 908 can be programmable and can be efficient for parallel workloads. The one or more GPUs 908 can use an extended Tensor instruction set in some examples. The one or more GPUs 908 can include one or more streaming microprocessors, each of which can contain an L1 cache (for example, an L1 cache with a minimum of 96 KB of memory), and two or more of the streaming microprocessors can share an L2 cache (for example, an L2 cache with a minimum of 512 KB of memory). In some embodiments, the one or more GPUs 908 can contain at least eight streaming microprocessors. The one or more GPUs 908 can use one or more application programming interfaces (APIs) for computation.Furthermore, the one or more GPUs 908 can use one or more parallel computing platforms and / or programming models (e.g. NVIDIA's CUDA).

[0149] The one or more GPUs 908 can be power-optimized for best performance in automotive and embedded applications. The one or more GPUs 908 can be manufactured, for example, on a FinFET field-effect transistor. However, this is not a limitation, and the one or more GPUs 908 can also be manufactured using other semiconductor manufacturing processes. Each streaming microprocessor can contain an array of mixed-precision processing cores, divided into multiple blocks. For example, and without limitation, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA TENSOR COREs for Deep Learning matrix arithmetic, an L0 instruction cache, a warp scheduler, a dispatch unit and / or a 64 KB register file.Furthermore, streaming microprocessors can include independent parallel integer and floating-point data paths to enable efficient execution of workloads with a mix of computations and addressing operations. Streaming microprocessors can include an independent thread scheduling function to allow fine-grained synchronization and cooperation between parallel threads. Streaming microprocessors can also include a combined L1 data cache and a shared memory unit to improve performance while simplifying programming.

[0150] The one or more GPUs 908 can include high-bandwidth memory (HBM) and / or a 16 GB HBM2 memory subsystem to provide a peak memory bandwidth of approximately 900 GB / second in some examples. In some examples, synchronous graphics random access memory (SGRAM), such as double-data-rate type five synchronous graphics random access memory (GDDR5), can be used in addition to or as an alternative to HBM memory.

[0151] The one or more GPUs 908 can incorporate a unified memory technology that includes access counters to enable more accurate migration of memory pages to the processor that accesses them most frequently, thereby improving the efficiency of memory areas shared by processors. In some examples, support for Address Translation Services (ATS) can be used so that the one or more GPUs 908 can directly access the page tables of the one or more CPUs 906. In such examples, if the Memory Management Unit (MMU) of the one or more GPUs 908 fails, an address translation request can be sent to the one or more CPUs 906.In response, the one or more CPUs 906 can search their page tables for the virtual-physical mapping for the address and send the translation back to the one or more GPUs 908. This unified memory technology thus enables a single, unified virtual address space for the memory of both the one or more CPUs 906 and the one or more GPUs 908, thereby simplifying the programming of the one or more GPUs 908 and the porting of applications to the one or more GPUs 908.

[0152] Additionally, the one or more GPUs 908 can contain an access counter that tracks the frequency of accesses by the one or more GPUs 908 to the memory of other processors. The access counter can help ensure that memory pages are moved to the physical memory of the processor that accesses them most frequently.

[0153] The one or more 904 SoCs can contain any number of 912 caches, including those described here. For example, the one or more 912 caches can contain an L3 cache that is available to both the one or more 906 CPUs and the one or more 908 GPUs (e.g., one that is connected to both the one or more 906 CPUs and the one or more 908 GPUs). The one or more 912 caches can contain a write-back cache that can track the states of the rows, for example, by using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). The L3 cache can be 4 MB or larger, depending on the implementation, although smaller cache sizes can also be used.

[0154] The one or more SoCs 904 can contain one or more Arithmetic Logic Units (ALUs) that can be used to perform processing related to one of the many tasks or operations of the Vehicle 900—such as DNN processing. Additionally, the one or more SoCs 904 can contain one or more Floating Point Units (FPUs)—or other mathematical or numerical coprocessors—for performing mathematical operations within the system. For example, the one or more SoCs 904 can contain one or more FPUs integrated as execution units into one or more CPUs 906 and / or one or more GPUs 908.

[0155] The one or more 904 SoCs can contain one or more 914 Accelerators (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, the one or more 904 SoCs can contain a hardware acceleration cluster, which may include optimized hardware accelerators and / or a large amount of on-chip memory. The large on-chip memory (e.g., 4 MB SRAM) can allow the hardware acceleration cluster to accelerate neural networks and other computations. The hardware acceleration cluster can be used in conjunction with the one or more 908 GPUs and offload some of the tasks performed by the one or more 908 GPUs (e.g., to free up more cycles of the one or more 908 GPUs for other tasks). The one or more 914 Accelerators can, for example, be used for specific workloads (e.g.,Perception, convolutional neural networks (CNNs), etc., are used that are stable enough to be suitable for acceleration. The term "CNN" as used here can include all types of CNNs, including region-based or regional convolutional neural networks (RCNNs) and fast RCNNs (e.g., for object detection).

[0156] The one or more Accelerators 914 (e.g., the Hardware Acceleration Cluster) can include a Deep Learning Accelerator (DLA). The one or more DLAs can include one or more Tensor Processing Units (TPUs) configured to provide an additional ten trillion operations per second for deep learning applications and inference. The TPUs can be accelerators configured and optimized to perform image processing functions (e.g., for CNNs, RCNNs, etc.). The one or more DLAs can also be optimized for a specific set of neural network types and floating-point operations, as well as for inference. The design of the one or more DLAs can deliver more performance per millimeter than a general-purpose GPU and far surpasses the performance of a CPU.The one or more TPUs can perform multiple functions, including a convolution function for a single instance that supports, for example, INT8, INT16 and FP16 data types for both features and weights, as well as post-processor functions.

[0157] One or more DLAs can quickly and efficiently run neural networks, especially CNNs, on processed or unprocessed data for a variety of functions, including, but not limited to: a CNN for object identification and detection using camera sensor data; a CNN for distance estimation using camera sensor data; a CNN for emergency vehicle detection and identification using microphone data; a CNN for facial recognition and vehicle owner identification using camera sensor data; and / or a CNN for security and / or protection-related events.

[0158] The one or more DLAs can execute any function of the one or more GPUs 908, and by using an inference accelerator, a developer can, for example, allocate either the one or more DLAs or the one or more GPUs 908 to each function. For example, the developer can concentrate the processing of CNNs and floating-point operations on the one or more DLAs and leave other functions to the one or more GPUs 908 and / or other accelerators 914.

[0159] The one or more Accelerators 914 (e.g., the Hardware Acceleration Cluster) can include a Programmable Vision Accelerator (PVA), which can also be referred to here as a Computer Vision Accelerator. The one or more PVAs can be designed and configured to accelerate computer vision algorithms for Advanced Driver Assistance Systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. The one or more PVAs can offer a balance between performance and flexibility. Each PVA can, for example, without limitation, include any number of Reduced Instruction Set Computer (RISC) cores, Direct Memory Access (DMA) cores, and / or any number of vector processors.

[0160] The RISC cores can interact with image sensors (e.g., the image sensors of one of the cameras described here), image signal processors, and / or the like. Each RISC core can contain any amount of memory. Depending on the implementation, the RISC cores can use any number of protocols. In some examples, the RISC cores can run a real-time operating system (RTOS). The RISC cores can be implemented with one or more integrated circuits, application-specific integrated circuits (ASICs), and / or memory devices. For example, the RISC cores can include an instruction cache and / or tightly coupled RAM.

[0161] The DMA allows components of the PVA(s) to access the system's main memory independently of the one or more CPUs. The DMA can support any number of features that serve to optimize the PVA, including, but not limited to, support for multidimensional and / or circular addressing. In some examples, the DMA can support up to six or more dimensions of addressing, which may include block width, block height, block depth, horizontal block stepping, vertical block stepping, and / or depth stepping.

[0162] Vector processors can be programmable processors designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing capabilities. In some examples, the PVA may contain a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, one or more DMA machines (e.g., two DMA machines), and / or other peripheral devices. The vector processing subsystem may operate as the primary processing machine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or working memory (e.g., VMEM).A VPU core can contain a digital signal processor, such as a single instruction, multiple data (SIMD) or a very long instruction word (VLIW). The combination of SIMD and VLIW can increase throughput and speed.

[0163] Each vector processor can contain an instruction cache and can be coupled to dedicated memory. Therefore, in some examples, each vector processor can be configured to operate independently of the others. In other examples, the vector processors contained in a particular PVA can be configured to use data parallelism. For example, in some embodiments, the multiple vector processors contained in a single PVA can execute the same computer vision algorithm, but on different regions of an image. In other examples, the vector processors contained in a particular PVA can simultaneously execute different computer vision algorithms on the same image, or even different algorithms on successive images or sections of an image.Among other things, any number of PVAs can be contained in the hardware acceleration cluster, and any number of vector processors can be contained in each of the PVAs. Furthermore, one or more PVAs can contain additional memory for error-correcting code (ECC) to increase the overall security of the system.

[0164] The one or more Accelerator 914 units (e.g., the hardware acceleration cluster) can include an on-chip computer vision network and SRAM to provide high-bandwidth, low-latency SRAM for the Accelerator 914. In some examples, the on-chip memory can include at least 4 MB of SRAM, consisting, for example, and without limitation, of eight field-configurable memory blocks accessible to both the PVA and the DLA. Each pair of memory blocks can include an Advanced Peripheral Bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory can be used. The PVA and the DLA can access the memory via a backbone, enabling high-speed memory access for both the PVA and the DLA.The backbone can include an on-chip computer vision network that connects the PVA and DLA to the main memory (e.g., using the APB).

[0165] The on-chip computer vision network can include an interface that, prior to the transmission of control signals / addresses / data, ensures that both the PVA and the DLA provide ready-to-use and valid signals. Such an interface can provide separate phases and channels for the transmission of control signals / addresses / data, as well as burst communication for continuous data transmission. This type of interface can conform to ISO 26262 or IEC 61508 standards, although other standards and protocols can also be used.

[0166] In some examples, one or more SoCs 904 can include a real-time ray tracing hardware accelerator as described in U.S. Patent Application No. 16 / 101,232, filed on August 10, 2018. The real-time ray tracing hardware accelerator can be used to quickly and efficiently determine the positions and extents of objects (e.g., within a world model) for generating real-time visualization simulations, for radar signal interpretation, for sound propagation synthesis and / or analysis, for simulating SONAR systems, for general wave propagation simulation, for comparison with LiDAR data for localization purposes, and / or for other functions and / or purposes. In some embodiments, one or more Tree Traversal Units (TTUs) can be used to perform one or more operations related to ray tracing.

[0167] The single or multiple Accelerator 914 (e.g., the hardware accelerator cluster) have a wide range of applications for autonomous driving. The PVA can be a programmable vision accelerator used for critical processing steps in ADAS and autonomous vehicles. The PVA's capabilities are well-suited to algorithmic domains requiring predictable processing with low power consumption and low latency. Thus, the PVA is well-suited for semi-dense or dense regular computations, even with small datasets, that demand predictable runtimes with low latency and low power consumption. In the context of autonomous vehicle platforms, PVAs are therefore designed to execute classic computer vision algorithms, as they are efficient at object detection and operate with integer mathematics.

[0168] According to one embodiment of the technology, the PVA is used, for example, to perform computer stereovision. In some examples, a semi-global matching-based algorithm can be used, although this is not intended as a limitation. Many applications for Level 3-5 autonomous driving require spontaneous motion estimation or stereo matching (e.g., structure of motion, pedestrian detection, lane detection, etc.). The PVA can perform computer stereovision on input from two monocular cameras.

[0169] In some examples, the PVA can be used to perform dense optical flow processing. This involves processing raw radar data (e.g., using a 4D Fast Fourier Transform) to provide processed radar data. In other examples, the PVA is used for time-of-flight depth processing, for example, by processing raw time-of-flight data to deliver processed time-of-flight data.

[0170] The DLA can be used to power any type of network to improve control and driving safety; this includes, for example, a neural network that outputs a confidence score for each object detection. Such a confidence score can be interpreted as a probability or as providing a relative "weighting" of each detection compared to other detections. This confidence score allows the system to make further decisions about which detections should be considered true positives and not false positives. For example, the system can set a confidence threshold and consider only those detections that exceed the threshold as true positives.In an automatic emergency braking (AEB) system, false positive detections would cause the vehicle to automatically initiate emergency braking, which is obviously undesirable. Therefore, only the safest detections should be considered as triggers for AEB. The DLA can employ a neural network to regress the confidence score. The neural network can use as input at least a subset of parameters, such as the dimensions of the boundary frame, the ground plane estimate (obtained, for example, from another subsystem), the output of the inertial measurement unit (IMU) sensor 966 correlated with the vehicle's orientation 900, distance, and 3D position estimates of the object obtained from the neural network and / or other sensors (e.g., one or more LiDAR sensors 964 or one or more radar sensors 960).

[0171] The one or more SoCs 904 can contain the one or more data stores 916 (e.g., main memory). The one or more data stores 916 can be on-chip main memory on the one or more SoCs 904, where neural networks can be stored to run on the GPU and / or the DLA. In some examples, the one or more data stores 916 can be large enough to store multiple instances of neural networks for redundancy and security. The one or more data stores 916 can include one or more L2 or L3 caches 912. The reference to the one or more data stores 916 can include a reference to the main memory allocated to the PVA, the DLA, and / or one or more other accelerators 914, as described herein.

[0172] The one or more 904 SoCs can contain one or more 910 processors (e.g., embedded processors). The one or more 910 processors can contain a boot and power management processor, which can be a dedicated processor and subsystem to handle boot power and management functions and the associated security enforcement. The boot and power management processor can be part of the boot sequence of the one or more 904 SoCs and can provide runtime power management services. The boot and power management processor can provide clock and voltage programming, support for system transitions to a low-power state, management of the thermals and temperature sensors of the one or more 904 SoCs, and / or management of the one or more 904 SoC power states.Each temperature sensor can be implemented as a ring oscillator whose output frequency is proportional to the temperature, and the one or more SoCs 904 can use the ring oscillators to detect the temperatures of the one or more CPUs 906, the one or more GPUs 908, and / or the one or more accelerators 914. If it is determined that the temperatures exceed a threshold, the boot and power management processor can enter a temperature fault routine and put the one or more SoCs 904 into a lower power state and / or put the vehicle 900 into a chauffeur-assured-stop mode (e.g., bring the vehicle 900 to a safe stop).

[0173] The one or more 910 processors can also include a number of embedded processors that can serve as an audio processing engine. The audio processing engine can be an audio subsystem that provides full hardware support for multi-channel audio across multiple interfaces and a wide and flexible range of audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core with a digital signal processor and dedicated RAM.

[0174] The one or more 910 processors can also include an always-on processor machine, which provides the necessary hardware functions to support low-power sensor management and wake-up from use cases. The always-on processor machine can include a processor core, tightly coupled RAM, supporting peripherals (such as timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0175] The one or more 910 processors can also include a security cluster machine, which contains a dedicated processor subsystem for the security management of automotive applications. The security cluster machine can include two or more processor cores, tightly coupled RAM, supporting peripherals (such as timers, an interrupt controller, etc.), and / or routing logic. In a security mode, the two or more cores can operate in lockstep mode, functioning as a single core with comparison logic that detects any differences between their operations.

[0176] The one or more 910 processors can also contain a real-time camera machine, which may include a dedicated processor subsystem for managing the real-time camera.

[0177] The one or more 910 processors may also include a high dynamic range signal processor, which may include an image signal processor, which is a hardware machine that is part of the camera processing pipeline.

[0178] The one or more 910 processors can include a video image compositor, which may be a processing block (e.g., implemented on a microprocessor) that implements video post-processing functions required by a video playback application to generate the final image for the player window. The video image compositor can perform lens distortion correction on the one or more 970 wide-angle cameras, the one or more 974 surround-view cameras, and / or on the sensors of the in-cabin surveillance camera. The in-cabin surveillance camera sensor is preferably monitored by a neural network running on a separate instance of the extended SoC and configured to detect events in the cabin and respond accordingly.A system in the cabin can lip-read to activate mobile service and make a call, dictate emails, change the destination, activate or change the infotainment system and vehicle settings, or enable voice-controlled internet browsing. Certain functions are only available to the driver when the vehicle is operating in autonomous mode and are otherwise deactivated.

[0179] The video image compositor can include enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, if there is motion in a video, the noise reduction weights the spatial information accordingly and reduces the impact of information provided by adjacent frames. If a frame or portion of a frame does not contain motion, the temporal noise reduction performed by the video image compositor can use information from the previous frame to reduce noise in the current frame.

[0180] The video image compositor can also be configured to perform stereo equalization of the input stereo lens images. Furthermore, the video image compositor can be used for user interface design when the operating system desktop is in use and the one or more GPUs 908 do not need to constantly render new surfaces. Even when the one or more GPUs 908 are powered on and actively performing 3D rendering, the video image compositor can be used to offload the load on the one or more GPUs 908, thus improving performance and responsiveness.

[0181] The one or more 904 SoCs can also include a serial camera interface with a Mobile Industry Processor Interface (MIPI) for receiving video and camera input, a high-speed interface, and / or a video input block that can be used for camera and related pixel input functions. The one or more 904 SoCs can also include one or more input / output controllers, one or more of which can be software-controlled and used for receiving I / O signals that are not assigned to a specific role.

[0182] The one or more 904 SoCs can also include a wide range of peripheral interfaces to allow communication with peripheral devices, audio codecs, power management, and / or other devices. The one or more 904 SoCs can be used to process data from cameras (e.g., via Gigabit Multimedia Serial Link and Ethernet), sensors (e.g., one or more 964 LiDAR sensors, one or more 960 radar sensors, etc., which can be connected via Ethernet), data from the 902 bus (e.g., vehicle speed, steering wheel position, etc.), and data from one or more 958 GNSS sensors (e.g., connected via Ethernet or CAN bus).Furthermore, the one or more SoCs 904 can contain dedicated high-performance mass storage controllers, which may contain their own DMA machines and can be used to offload routine data management tasks from the one or more CPUs 906.

[0183] The single or multiple 904 SoCs can form an end-to-end platform with a flexible architecture spanning automation levels 3-5, thereby providing a comprehensive functional safety architecture that supports and efficiently utilizes computer vision and ADAS techniques for diversity and redundancy, and provides a platform for a flexible, reliable driving software stack along with deep learning tools. The single or multiple 904 SoCs can be faster, more reliable, and even more energy- and space-efficient than conventional systems. For example, the single or multiple 914 accelerators, in combination with the single or multiple 906 CPUs, the single or multiple 908 GPUs, and the single or multiple 916 data stores, can form a fast, efficient platform for autonomous vehicles of levels 3-5.

[0184] This technology thus offers capabilities and functions that cannot be achieved by conventional systems. For example, computer vision algorithms can be run on CPUs that can be configured using a high-level programming language, such as C, to execute a variety of processing algorithms on a wide range of visual data. However, CPUs are often unable to meet the performance requirements of many computer vision applications, such as execution time and power consumption. In particular, many CPUs are unable to execute complex object detection algorithms in real time, which is a requirement for in-vehicle ADAS applications and a prerequisite for practical Level 3-5 autonomous vehicles.

[0185] Unlike conventional systems, the technology described herein, by providing a CPU complex, a GPU complex, and a hardware acceleration cluster, enables the simultaneous and / or sequential execution of multiple neural networks and the combination of their results to allow for Level 3-5 autonomous driving functionality. For example, a CNN running on the DLA or the dGPU (e.g., one or more GPUs 920) can include text and word recognition, allowing the supercomputer to read and understand traffic signs, including signs for which the neural network has not been specifically trained. The DLA can further include a neural network capable of identifying and interpreting the sign, providing a semantic understanding, and passing this semantic understanding to the path planning modules running on the CPU complex.

[0186] Another example is that multiple neural networks can run simultaneously, as required for driving at levels 3, 4, or 5. For instance, a warning sign reading "Caution: Flashing lights indicate black ice" accompanied by an electric light can be interpreted independently or jointly by several neural networks. The sign itself can be identified as a traffic sign by a first neural network (e.g., a trained one), while the text "Flashing lights indicate black ice" can be interpreted by a second neural network, which then informs the vehicle's path planning software (preferably running on the CPU) that the presence of black ice indicates the presence of flashing lights.The turn signal can be identified across multiple images by a third neural network, which informs the vehicle's path planning software about the presence (or absence) of turn signals. All three neural networks can run simultaneously, for example, within the DLA and / or on one or more GPUs 908.

[0187] In some examples, a CNN for facial recognition and vehicle owner identification can use data from camera sensors to identify the presence of an authorized driver and / or owner of the Vehicle 900. The always-on sensor processing unit can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and to disable the vehicle in security mode when the owner leaves. In this way, the one or more SoCs 904 provide security against theft and / or carjacking.

[0188] In another example, a CNN for detecting and identifying emergency vehicles can use data from microphones 996 to detect and identify emergency vehicle sirens. Unlike conventional systems that use general classifiers to detect sirens and manually extract features, the one or more SoCs 904 use the CNN to classify environmental and urban sounds as well as visual data. In a preferred embodiment, the CNN running on the DLA is trained to detect the relative approach speed of the emergency vehicle (e.g., by using the Doppler effect). The CNN can also be trained to identify emergency vehicles specific to the local area in which the vehicle is operating, as identified by one or more GNSS sensors 958.For example, the CNN will attempt to detect European sirens when operating in Europe, and when operating in the United States, the CNN will attempt to identify only North American sirens. Once an emergency vehicle is detected, a controller can be used to execute an emergency vehicle safety routine, slowing the vehicle down, pulling over to the side of the road, parking the vehicle, and / or letting the vehicle idle, using the 962 ultrasonic sensors, until one or more emergency vehicles pass.

[0189] The vehicle can contain one or more CPUs 918 (e.g., one or more discrete CPUs or one or more dCPUs) that can be coupled to the one or more SoCs 904 via a high-speed connection (e.g., PCIe). The CPUs 918 can, for example, contain an x86 processor. The CPUs 918 can be used, for example, to perform a variety of functions, including reconciling potentially inconsistent results between ADAS sensors and the one or more SoCs 904 and / or monitoring the status and health of the one or more Controllers 936 and / or the Infotainment SoC 930.

[0190] The Vehicle 900 can contain one or more GPUs 920 (e.g., one or more discrete GPUs or one or more dGPUs) that can be coupled to the one or more SoCs 904 via a high-speed connection (e.g., NVIDIA's NVLINK). The one or more GPUs 920 can provide additional artificial intelligence capabilities, such as running redundant and / or distinct neural networks, and can be used to train and / or update neural networks based on input (e.g., sensor data) from sensors in the Vehicle 900.

[0191] The vehicle 900 can also include the network interface 924, which can contain one or more wireless antennas 926 (e.g., one or more wireless antennas for different communication protocols, such as a cellular antenna, a Bluetooth antenna, etc.). The network interface 924 can be used to allow a wireless connection via the internet to the cloud (e.g., to one or more servers 978 and / or other network devices), to other vehicles, and / or to computing devices (e.g., client devices of passengers). To communicate with other vehicles, a direct connection between the two vehicles and / or an indirect connection (e.g., via networks and the internet) can be established. Direct connections can be established via vehicle-to-vehicle communication.Vehicle-to-vehicle communication can provide the Vehicle 900 with information about vehicles in its vicinity (e.g., vehicles in front of, beside, and / or behind the Vehicle 900). This functionality can be part of a cooperative adaptive cruise control function of the Vehicle 900.

[0192] The 924 network interface can include a system-on-a-chip (SoC) that provides modulation and demodulation functions, enabling one or more 936 controllers to communicate over wireless networks. The 924 network interface can include a high-frequency (RF) front end for up-conversion from baseband to RF and down-conversion from RF to baseband. Frequency conversions can be performed using known methods and / or superheterodyne methods. In some examples, the RF front-end functionality can be provided by a separate chip. The network interface can include wireless functionality for communication via LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0193] The vehicle 900 may further include one or more data storage devices 928, which may be located outside the chip (e.g., outside the SoCs 904). The one or more data storage devices 928 may contain one or more memory elements, including RAM, SRAM, DRAM, VRAM, flash, hard disks, and / or other components and / or devices capable of storing at least one bit of data.

[0194] The Vehicle 900 can also include one or more GNSS Sensors 958. The one or more GNSS Sensors 958 (e.g., GPS, supported GPS sensors, differential GPS (DGPS) sensors, etc.) assist with mapping, perception, occupancy grid creation, and / or path planning. Any number of GNSS Sensors 958 can be used, including, for example, and without limitation, a GPS unit that uses a USB connection with an Ethernet-to-serial (RS-232) bridge.

[0195] The vehicle 900 can also include one or more RADAR sensors 960. The vehicle 900 can use the one or more RADAR sensors 960 to detect vehicles at long range, even in darkness and / or adverse weather conditions. The functional safety level of the RADAR can be ASIL B. The one or more RADAR sensors 960 can use the CAN bus and / or the 902 bus (e.g., for transmitting data generated by the one or more RADAR sensors 960) for control and access to object tracking data, with some examples using Ethernet for access to the raw data. A variety of RADAR sensor types can be used. The one or more RADAR sensors 960 can be suitable for front, rear, and side RADAR applications without restriction. In some examples, one or more pulse-Doppler RADAR sensors are used.

[0196] The single or multiple RADAR 960 sensors can incorporate various configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range side coverage, etc. In some cases, long-range RADAR can be used for adaptive cruise control. Long-range RADAR systems can provide a wide field of view, achieved through two or more independent scans, for example, within a range of 250 m. The single or multiple RADAR 960 sensors can assist in distinguishing between stationary and moving objects and can be used by ADAS systems for emergency braking assistance and forward collision warning. Long-range RADAR sensors can incorporate a monostatic multimodal RADAR with multiple (e.g., six or more) fixed RADAR antennas and a high-speed CAN and FlexRay interface.In an example with six antennas, the four central antennas can generate a focused beam pattern designed to detect the area around vehicle 900 at higher speeds with minimal interference from traffic in adjacent lanes. The other two antennas can expand the field of view, enabling the rapid detection of vehicles entering or exiting vehicle 900's lane.

[0197] Medium-range radar systems, for example, can have a range of up to 960 m (front) or 80 m (rear) and a field of view of up to 42 degrees (front) or 950 degrees (rear). Short-range radar systems can include, among other things, radar sensors designed for installation at both ends of the rear bumper. When such a radar sensor system is installed at both ends of the rear bumper, it can generate two beams that continuously monitor the blind spot behind and to the sides of the vehicle.

[0198] Short-range radar systems can be used in an ADAS system for blind spot detection and / or as a lane change assistant.

[0199] The vehicle 900 can also include one or more ultrasonic sensors 962. The one or more ultrasonic sensors 962, which can be mounted on the front, rear, and / or sides of the vehicle 900, can be used for parking assistance and / or for creating and updating an occupancy grid. A variety of ultrasonic sensors 962 can be used, and different ultrasonic sensors 962 can be used for different detection ranges (e.g., 2.5 m, 4 m). The one or more ultrasonic sensors 962 can operate with functional safety levels of ASIL B.

[0200] The vehicle 900 can contain one or more LiDAR sensors 964. The one or more LiDAR sensors 964 can be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The one or more LiDAR sensors 964 can meet the functional safety level ASIL B. In some examples, the vehicle 900 can contain multiple LiDAR sensors 964 (e.g., two, four, six, etc.) that can use Ethernet (e.g., to deliver data to a Gigabit Ethernet switch).

[0201] In some examples, one or more LiDAR sensors 964 may be able to provide a list of objects and their distances for a 360-degree field of view. Commercially available LiDAR sensors 964 may, for example, have a specified range of approximately 900 m, with an accuracy of 2 cm to 3 cm and support for a 900 Mbit / s Ethernet connection. In some examples, one or more non-protruding LiDAR sensors 964 may be used. In such examples, the one or more LiDAR sensors 964 may be implemented as a small device that can be embedded in the front, rear, sides, and / or corners of the vehicle 900. In such examples, one or more LiDAR 964 sensors can provide a horizontal field of view of up to 120 degrees and a vertical field of view of up to 35 degrees, with a range of 200 m, even with objects of low reflectivity.The one or more front-mounted LiDAR 964 sensors can be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0202] In some examples, LiDAR technologies, such as 3D flash LiDAR, can also be used. 3D flash LiDAR uses a laser pulse as a transmission source to illuminate the vehicle's surroundings up to approximately 200 m. A flash LiDAR unit contains a sensor that records the travel time of the laser pulse and the reflected light at each pixel, which in turn corresponds to the distance between the vehicle and the objects. Flash LiDAR can enable the generation of highly accurate and distortion-free images of the surroundings with each laser pulse. In some examples, four flash LiDAR sensors can be used, one on each side of the vehicle. Available 3D flash LiDAR systems include a solid-state 3D focal plane array LiDAR camera that contains no moving parts other than a fan (e.g., a non-scanning LiDAR device).The flash LiDAR device can use a 5-nanosecond pulse of a Class I (eye-safe) laser per frame and capture the reflected laser light in the form of 3D distance point clouds and co-registered intensity data. By using flash LiDAR, and because flash LiDAR is a solid-state device with no moving parts, the single or multiple LiDAR sensors can be less susceptible to motion blur, vibration, and / or shock.

[0203] The vehicle may also contain one or more IMU sensors 966. In some examples, the one or more IMU sensors 966 may be located in the center of the rear axle of the vehicle 900. The one or more IMU sensors 966 may, for example, and without limitation, contain one or more accelerometers, one or more magnetometers, one or more gyroscopes, one or more magnetic compasses, and / or other sensor types. In some examples, such as six-axis applications, the one or more IMU sensors 966 may contain accelerometers and gyroscopes, while in nine-axis applications, the one or more IMU sensors 966 may contain accelerometers, gyroscopes, and magnetometers.

[0204] In some embodiments, the one or more IMU sensors 966 can be implemented as a miniaturized, high-performance GPS-Aided Inertial Navigation System (GPS / INS) that combines inertial sensors of a microelectromechanical system (MEMS), a highly sensitive GPS receiver, and advanced Kalman filter algorithms to provide estimates of position, velocity, and orientation. Thus, in some examples, the one or more IMU sensors 966 can allow the vehicle 900 to estimate its course without requiring input from a magnetic sensor by directly observing and correlating velocity changes from the GPS with the one or more IMU sensors 966. In some examples, the one or more IMU sensors 966 and the one or more GNSS sensors 958 can be combined in a single integrated unit.

[0205] The vehicle may contain one or more microphones 996, which are mounted in and / or around the vehicle 900. The one or more microphones 996 may be used, among other things, for the detection and identification of emergency vehicles.

[0206] The vehicle may also include any number of camera types, including one or more stereo cameras 968, one or more wide-angle cameras 970, one or more infrared cameras 972, one or more surround-view cameras 974, one or more long-range and / or medium-range cameras 998, and / or other camera types. The cameras can be used to capture image data around the entire periphery of the vehicle 900. The types of cameras used depend on the embodiment and requirements of the vehicle 900, and any combination of camera types can be used to ensure the necessary coverage around the vehicle 900. Furthermore, the number of cameras can vary depending on the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or any other number of cameras.The cameras can, for example and without limitation, support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet. Each of the one or more cameras is described here with reference to... Fig. 9A and Fig. 9B is described in more detail.

[0207] The vehicle 900 may also include one or more vibration sensors 942. The one or more vibration sensors 942 can measure vibrations of vehicle components, such as one or more axles. For example, changes in vibration may indicate a change in the road surface. In another example, if two or more vibration sensors 942 are used, the differences between the vibrations can be used to determine friction or slippage on the road surface (e.g., if the difference in vibration is between a driven axle and a freely rotating axle).

[0208] The vehicle 900 may include an ADAS system 938. In some examples, the ADAS system 938 may include a SoC. The ADAS system 938 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward crash warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keep assist (LKA), blind spot warning (BSW), rear cross-traffic warning (RCTW), collision warning systems (CWS), lane centering (LC), and / or other features and functions.

[0209] The ACC systems can use one or more radar sensors (960), one or more LiDAR sensors (964), and / or one or more cameras. The ACC systems can include longitudinal and / or lateral ACC. Longitudinal ACC monitors and controls the distance to the vehicle immediately in front of the vehicle (900) and automatically adjusts the vehicle speed to maintain a safe distance from vehicles ahead. Lateral ACC performs distance control and advises the vehicle (900) to change lanes if necessary. Lateral ACC interacts with other ADAS applications, such as LCA and CWS.

[0210] The CACC uses information from other vehicles, which can be received via the network interface 924 and / or the one or more wireless antennas 926 from other vehicles via a wireless connection or indirectly via a network connection (e.g., via the internet). Direct connections can be provided via a vehicle-to-vehicle (V2V) communication link, while indirect connections can be an infrastructure-to-vehicle (I2V) communication link. In general, the V2V communication concept provides information about the vehicles immediately ahead (e.g., vehicles directly in front of the vehicle 900 and in the same lane), while the I2V communication concept provides information about traffic further ahead. CACC systems can incorporate one or both of the I2V and V2V information sources.Given the information about the vehicles ahead of vehicle 900, CACC can be more reliable and has the potential to improve traffic flow and reduce congestion on the road.

[0211] FCW systems are designed to warn the driver of a hazard, allowing them to take corrective action. FCW systems utilize a forward-facing camera and / or one or more RADAR 960 sensors coupled with a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically connected to the driver feedback system, such as a display, speaker, and / or vibrating component. FCW systems can provide a warning in the form of an audible signal, a visual warning, a vibration, and / or a rapid braking pulse.

[0212] AEB systems detect an impending head-on collision with another vehicle or object and can automatically apply the brakes if the driver does not take corrective action within a specific time or distance parameter. AEB systems can use one or more forward-facing cameras and / or one or more RADAR 960 sensors coupled with a dedicated processor, DSP, FPGA, and / or ASIC. When the AEB system detects a hazard, it typically first warns the driver so they can take corrective action to avoid the collision; if the driver does not take corrective action, the AEB system can automatically apply the brakes to prevent or at least mitigate the effects of the predicted collision. AEB systems may incorporate techniques such as dynamic brake assist and / or emergency braking for an impending collision.

[0213] Lane Departure Warning (LDW) systems provide visual, audible, and / or tactile warnings, such as steering wheel or seat vibrations, to alert the driver if the vehicle crosses lane markings. An LDW system will not activate if the driver indicates an intentional lane departure by using a turn signal. LDW systems may utilize forward-facing cameras coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically connected to the feedback signal for the driver, such as a display, speaker, and / or vibrating component.

[0214] LKA systems are a variant of LDW systems. LKA systems provide steering or braking inputs to correct vehicle 900 if the vehicle 900 begins to leave its lane.

[0215] Blind Spot Warning (BSW) systems detect and warn the driver of vehicles in the car's blind spot. BSW systems can issue a visual, audible, and / or tactile warning signal to indicate that merging into or changing lanes is unsafe. The system can issue an additional warning if the driver activates a turn signal. BSW systems can use one or more rear-facing cameras and / or radar sensors coupled with a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to the feedback to the driver, such as a display, speaker, and / or vibrating component.

[0216] RCTW systems can provide visual, audible, and / or tactile alerts when an object is detected outside the reversing camera's field of view while the vehicle is reversing. Some RCTW systems incorporate AEB to ensure the vehicle's brakes are applied to prevent a collision. RCTW systems can utilize one or more rear-facing radar sensors coupled with a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically connected to the driver for feedback, such as a display, speaker, and / or vibrating component.

[0217] Conventional ADAS systems can produce false positives, which, while annoying and distracting for the driver, are generally not catastrophic because the ADAS systems warn the driver and give them the opportunity to decide whether a safety issue truly exists and to act accordingly. In an autonomous vehicle 900, however, the vehicle 900 itself must decide, in the event of conflicting results, whether to follow the result from a primary computer or a secondary computer (e.g., a first controller 936 or a second controller 936). In some embodiments, the ADAS system 938 can, for example, be a backup and / or secondary computer that provides information about perception to a rationality module of the backup computer.The backup computer rationality monitor can run redundant, diverse software on hardware components to detect errors in perception and dynamic driving tasks. The outputs of the ADAS system 938 can be provided to a monitoring MCU. If the outputs of the primary and secondary computers conflict, the monitoring MCU must determine how to resolve the conflict to ensure safe operation.

[0218] In some examples, the primary computer can be configured to provide the monitoring MCU with a confidence score indicating its confidence in the chosen outcome. If the confidence score exceeds a threshold, the monitoring MCU can follow the primary computer's instruction, regardless of whether the secondary computer returns a conflicting or inconsistent result. If the confidence score does not reach the threshold and the primary and secondary computers display different results (e.g., conflicting results), the monitoring MCU can mediate between the computers to determine the appropriate outcome.

[0219] The monitoring MCU can be configured to run one or more neural networks trained and configured to determine, based on the outputs of the primary and secondary computers, the conditions under which the secondary computer will trigger false alarms. This allows the one or more neural networks in the monitoring MCU to learn when the secondary computer's output can be trusted and when it cannot. For example, if the secondary computer is a radar-based FCW system, a neural network in the monitoring MCU can learn to trigger an alarm when the FCW system identifies metallic objects that do not actually pose a threat, such as a drain grate or manhole cover.Similarly, if the secondary computer is a camera-based lane departure warning (LDW) system, a neural network in the supervising MCU can learn to override the LDW system when cyclists or pedestrians are present and leaving the lane is indeed the safest maneuver. In embodiments containing one or more neural networks running on the supervising MCU, the supervising MCU can include at least one DLA or GPU suitable for executing the one or more neural networks with associated memory. In preferred embodiments, the supervising MCU can include and / or be contained as a component of the one or more SoCs 904.

[0220] In other examples, the ADAS System 938 can include a secondary computer that executes the ADAS functionality according to the classical rules of computer vision. Thus, the secondary computer can use classical computer vision rules (if-then), and the presence of one or more neural networks in the monitoring MCU can improve reliability, safety, and performance. For example, the diverse implementation and intentional non-identity make the overall system more fault-tolerant, especially to errors caused by software (or software-hardware interfaces).For example, if a software bug or error occurs in the software on the primary computer and the non-identical software code on the secondary computer produces the same overall result, the monitoring MCU can have greater confidence that the overall result is correct and that the bug in the software or hardware on the primary computer does not cause a significant error.

[0221] In some examples, the output of the ADAS system 938 can be fed into the perception block of the primary computer and / or the dynamic driving task block of the primary computer. For example, if the ADAS system 938 displays a frontal collision warning due to an object directly in front of the vehicle, the perception block can use this information in object identification. In other examples, the secondary computer may have its own trained neural network, thus reducing the risk of false positives, as described herein.

[0222] The Vehicle 900 may also include the Infotainment SoC 930 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system may not actually be an SoC and may contain two or more discrete components. The Infotainment SoC 930 may include a combination of hardware and software that can be used to provide the Vehicle 900 with audio (e.g., music, a personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and / or information services (e.g., navigation systems, rear parking sensors, a radio data system, vehicle-related information such as fuel level, total distance traveled, brake fluid level, oil level, door open / close status, air filter information, etc.).The Infotainment SoC 930 can include, for example, radios, record players, navigation systems, video players, USB and Bluetooth connectivity, car computers, in-car entertainment, Wi-Fi, steering wheel audio controls, hands-free systems, a head-up display (HUD), an HMI display 934, a telematics device, a control panel (e.g., for controlling and / or interacting with various components, functions, and / or systems), and / or other components. The Infotainment SoC 930 can also be used to provide information (e.g., visual and / or audible) to one or more vehicle users, such as information from the ADAS system 938, autonomous driving information such as planned vehicle maneuvers, road layouts, environmental information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0223] The Infotainment SoC 930 can include GPU functionality. The Infotainment SoC 930 can communicate with other devices, systems, and / or components of the Vehicle 900 via the Bus 902 (e.g., CAN bus, Ethernet, etc.). In some examples, the Infotainment SoC 930 can be coupled with a monitoring MCU so that the Infotainment System's GPU can perform some self-driving functions if one or more of the Primary Controllers 936 (e.g., the Vehicle 900's primary and / or backup computers) fail. In such an example, the Infotainment SoC 930 can put the Vehicle 900 into a chauffeur-to-safe-stop mode, as described here.

[0224] The vehicle 900 may also include an instrument cluster 932 (e.g., a digital instrument cluster, an electronic instrument cluster, a digital instrument panel, etc.). The instrument cluster 932 may include a controller and / or supercomputer (e.g., a discrete controller or supercomputer). The instrument cluster 932 may contain a number of instruments, such as a speedometer, fuel gauge, oil pressure gauge, tachometer, odometer, turn signals, shift position indicator, seat belt warning light(s), parking brake warning light(s), engine malfunction light(s), airbag system (SRS) information, lighting controls, safety system controls, navigation information, etc. In some examples, information from the infotainment SoC 930 and the instrument cluster 932 may be displayed and / or shared. Thus, the instrument cluster 932 may be included as part of the infotainment SoC 930, or vice versa.

[0225] Fig. Figure 9D shows a system diagram for the communication between one or more cloud-based servers and the exemplary autonomous vehicle 900. Fig. 9A, according to some embodiments of the present disclosure; The system 976 may include one or more servers 978, one or more networks 990, and the vehicles, including the vehicle 900. The server(s) 978 may include multiple GPUs 984(A)-984(H) (here collectively referred to as GPUs 984), PCIe switches 982(A)-982(D) (here collectively referred to as PCIe switches 982), and / or CPUs 980(A)-980(B) (here collectively referred to as CPUs 980). The GPUs 984, the CPUs 980, and the PCIe switches may be interconnected by high-speed links, such as, but not limited to, NVIDIA's NVLink interfaces 988 and / or PCIe links 986. In some examples, the GPUs 984 are connected via NVLink and / or NVSwitch SoC, and the GPUs 984 and the PCIe switches 982 are connected via PCIe connections.Although eight GPUs 984, two CPUs 980, and two PCIe switches are illustrated, this should not be interpreted as a limitation. Depending on the configuration, each Server 978 can contain any number of GPUs 984, CPUs 980, and / or PCIe switches. For example, one or more Server 978s can each contain eight, sixteen, thirty-two, and / or more GPUs 984.

[0226] The one or more servers 978 can receive image data from the vehicles via the one or more networks 990. This image data is representative of images showing unexpected or changed road conditions, such as recently started roadworks. The one or more servers 978 can transmit neural networks 992, updated neural networks 992, and / or map information 994 to the vehicles via the one or more networks 990. This map information contains information about traffic and road conditions. The map information updates 994 can include updates for the HD map 922, such as information about construction sites, potholes, detours, floods, and / or other obstacles.In some examples, the neural networks 992, the updated neural networks 992 and / or the map information 994 may result from new training and / or experience represented in the data received from any number of vehicles in the environment, and / or may be based on training performed in a data center (e.g. using one or more servers 978 and / or other servers).

[0227] One or more Server 978s can be used to train machine learning models (e.g., neural networks) based on training data. The training data can be generated using the vehicles and / or in a simulation (e.g., using a game machine). In some examples, the training data is tagged (e.g., if the neural network benefits from supervised learning) and / or subjected to other preprocessing, while in other examples, the training data is not tagged and / or preprocessed (e.g., if the neural network does not require supervised learning).Training can be performed using one or more classes of machine learning techniques, including, but not limited to, classes such as: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, federated learning, transfer learning, feature learning (including principal component and cluster analysis), multilinear subspace learning, diverse learning, representational learning (including substitute dictionary learning), rule-based machine learning, anomaly detection, and all variants or combinations thereof. Once the machine learning models are trained, they can be used by the vehicles (e.g., transmitted to the vehicles via one or more networks 990) and / or the machine learning models can be used by one or more servers 978 for remote monitoring of the vehicles.

[0228] In some examples, one or more Server 978s can receive data from the vehicles and apply that data to advanced neural networks in real time for intelligent, real-time inference. The one or more Server 978s can include deep learning supercomputers and / or dedicated AI computers powered by GPUs 984, such as NVIDIA's DGX and DGX Station machines. However, in some examples, the one or more Server 978s can include a deep learning infrastructure that uses only CPU-powered data centers.

[0229] The deep learning infrastructure of one or more Server 978 systems can perform fast, real-time inference and can use this capability to assess and verify the state of the processors, software, and / or associated hardware in the Vehicle 900. For example, the deep learning infrastructure can receive periodic updates from the Vehicle 900, such as a sequence of images and / or objects that the Vehicle 900 has located within that sequence (e.g., via computer vision and / or other machine learning object classification techniques).The deep learning infrastructure can run its own neural network to identify the objects and compare them with the objects identified by the vehicle 900. If the results do not match and the infrastructure concludes that the AI ​​in the vehicle 900 is not working correctly, one or more servers 978 can send a signal to the vehicle 900, instructing a fail-safe computer in the vehicle 900 to take control, notify the passengers, and perform a safe parking maneuver.

[0230] For inference, one or more Server 978s can include GPUs 984s and one or more programmable inference accelerators (e.g., NVIDIA's TensorRT). The combination of GPU-driven servers and inference accelerators can enable real-time responsiveness. In other scenarios, such as when performance is less critical, servers powered by CPUs, FPGAs, and other processors can be used for inference. EXAMPLE CALCULATION DEVICE

[0231] Fig. Figure 10 is a block diagram of an exemplary computing device 1000 suitable for use in implementing some embodiments of the present disclosure. The computing device 1000 may include a connection system 1002 that directly or indirectly couples the following devices: main memory 1004, one or more central processing units (CPUs) 1006, one or more graphics processing units (GPUs) 1008, a communication interface 1010, input / output (I / O) ports 1012, input / output components 1014, a power supply 1016, one or more presentation components 1018 (e.g., display(s)), and one or more logic units 1020. In at least one embodiment, the one or more computing devices 1000 may comprise one or more virtual machines (VMs), and / or each of the components thereof may comprise virtual components (e.g., virtual hardware components).As non-restrictive examples, one or more of the GPUs 1008 can comprise one or more vGPUs, one or more of the CPUs 1006 can comprise one or more vCPUs, and / or one or more of the logic units 1020 can comprise one or more virtual logic units. Thus, a computing device 1000 can contain discrete components (e.g., a complete GPU allocated to computing device 1000), virtual components (e.g., a portion of a GPU allocated to computing device 1000), or a combination thereof.

[0232] Although the various blocks of Fig. Where components 10 are shown as connected via the connection system 1002, this is not intended as a limitation and serves only for clarity. In some embodiments, for example, a presentation component 1018, such as a display device, can be considered an I / O component 1014 (e.g., if the display is a touchscreen). As another example, the CPUs 1006 and / or GPUs 1008 can contain main memory (e.g., the main memory 1004 can represent a storage device in addition to the main memory of the GPUs 1008, the CPUs 1006, and / or other components). Thus, the computing device of Fig. 10 is for illustrative purposes only. No distinction is made between categories such as "workstation", "server", "laptop", "desktop", "tablet", "client device", "mobile device", "handheld device", "game console", "electronic control unit (ECU)", "virtual reality system" and / or other device or system types, as all are within the scope of protection of the computing device of Fig. 10 are being considered.

[0233] The connection system 1002 can represent one or more connections or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The connection system 1002 can include one or more bus or connection types, such as an Industry Standard Architecture (ISA) bus, an Extended ISA bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI Express (PCIe) bus, and / or another type of bus or connection. In some embodiments, there are direct connections between components. For example, the CPU 1006 can be directly connected to the RAM 1004. Furthermore, the CPU 1006 can be directly connected to the GPU 1008.In a direct or point-to-point connection between components, the connection system 1002 can include a PCIe link to establish the connection. In these examples, a PCI bus does not need to be included in the computing device 1000.

[0234] The main memory 1004 can contain a variety of computer-readable media. Computer-readable media can be any available media that the computing device 1000 can access. Computer-readable media can include both volatile and non-volatile media, as well as removable and non-removable media. For example, and without limitation, computer-readable media can include computer storage media and communication media.

[0235] Computer storage media can include both volatile and non-volatile media, and / or removable and non-removable media, implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, and / or other data types. For example, main memory can store computer-readable instructions (e.g., representing one or more programs and / or one or more program elements, such as an operating system). Computer storage media can include, but are not limited to, RAM, ROM, EEPROM, flash memory, or other storage technologies; CD-ROM, Digital Versatile Discs (DVDs), or other optical disk storage; magnetic cartridges, magnetic tapes, magnetic disk storage, or other magnetic storage devices; or any other medium that can be used to store the desired information and that the computing device can access.As used here, computer storage media do not inherently contain signals.

[0236] Computer storage media can embody computer-readable instructions, data structures, program modules, and / or other data types in a modulated data signal, such as a carrier wave or other transport mechanism, and include any media for transmitting information. The term "modulated data signal" can refer to a signal in which one or more of its properties are set or modified to encode information within the signal. Computer storage media can include, but are not limited to, wired media, such as a wired network or a direct-wired connection, and wireless media, such as acoustic, RF, infrared, and other wireless media. Combinations of the above should also be included in the scope of protection of the computer-readable media.

[0237] The one or more CPUs 1006 can be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1000 in order to perform one or more of the procedures and / or processes described herein. The one or more CPUs 1006 can each contain one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of processing a multitude of software threads simultaneously. The one or more CPUs 1006 can contain any type of processor and can contain different types of processors depending on the type of computing device 1000 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers).Depending on the type of computing device 1000, the processor can be, for example, an Advanced RISC Machine (ARM) processor implemented with Reduced Instruction Set Computing (RISC), or an x86 processor implemented with Complex Instruction Set Computing (CISC). The computing device 1000 can contain one or more CPUs 1006, in addition to one or more microprocessors or additional coprocessors, such as mathematical coprocessors.

[0238] In addition to or as an alternative to the one or more CPUs 1006, the one or more GPUs 1008 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1000 in order to perform one or more of the procedures and / or processes described herein. One or more of the GPUs 1008 can be an integrated GPU (e.g., with one or more of the CPUs 1006) and / or one or more of the GPUs 1008 can be a discrete GPU. In embodiments, one or more of the GPUs 1008 can be a co-processor of one or more of the CPUs 1006. The one or more GPUs 1008 can be used by the computing device 1000 to render graphics (e.g., 3D graphics) or to perform general-purpose computing. The one or more GPUs 1008 can be used, for example, for general-purpose computing on GPUs (GPGPU).The one or more GPUs 1008 can contain hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. The one or more GPUs 1008 can generate pixel data for output images in response to rendering commands (e.g., rendering commands from the one or more CPUs 1006 received via a host interface). The one or more GPUs 1008 can include graphics memory, such as display memory, for storing pixel data or other suitable data, such as GPGPU data. The display memory can be included as part of the main memory 1004. The one or more GPUs 1008 can contain two or more GPUs operating in parallel (e.g., via a link). The link can connect the GPUs directly (e.g., using NVLINK) or connect them via a switch (e.g., using NVSwitch).When combined, each GPU can generate 1008 pixel data or GPGPU data for different sections of an output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU can have its own dedicated memory or share memory with other GPUs.

[0239] In addition to or as an alternative to the one or more CPUs 1006 and / or the one or more GPUs 1008, the one or more logic units 1020 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1000 to perform one or more of the methods and / or processes described herein. In embodiments, the one or more CPUs 1006, the one or more GPUs 1008, and / or the one or more logic units 1020 may discretely or jointly execute any combination of the methods, processes, and / or sections thereof. One or more of the logic units 1020 may be part of and / or integrated into one or more of the CPUs 1006 and / or one or more of the GPUs 1008, and / or one or more of the logic units 1020 may be discrete components or otherwise separate from the CPUs 1006 and / or the GPUs 1008.In embodiments, one or more of the logic units 1020 can be a co-processor of one or more of the CPUs 1006 and / or one or more of the GPUs 1008.

[0240] Examples of one or more logic units 1020 include one or more processing cores and / or components thereof, such as data processing units (DPUs), tensor cores (TCs), tensor processing units (TPUs), pixel visual cores (PVCs), vision processing units (VPUs), graphics processing clusters (GPCs), texture processing clusters (TPCs), streaming multiprocessors (SMs), tree traversal units (TTUs), artificial intelligence accelerators (AIAs), deep learning accelerators (DLAs), arithmetic logic units (ALUs), and application-specific integrated circuits. (Application-Specific Integrated Circuits, ASICs), Floating Point Units (FPUs),Input / output (I / O) elements, peripheral component interconnect (PCI) or PCI Express (PCIe) elements, and / or similar.

[0241] The 1010 communication interface can contain one or more receivers, transmitters, and / or transceivers that allow the 1000 computing device to communicate with other computing devices over an electronic network, including wired and / or wireless communication. The 1010 communication interface can include components and functions that allow communication over a variety of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., Ethernet or InfiniBand communication), low-power wide-area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet.In one or more embodiments, the one or more logic units 1020 and / or the communication interface 1010 may contain one or more data processing units (DPUs) to transfer data received via a network and / or via the connection system 1002 directly to one or more GPUs 1008 (e.g., a working memory thereof).

[0242] The I / O ports 1012 allow the computing device 1000 to be logically coupled with other devices, including the I / O components 1014, one or more presentation components 1018, and / or other components, some of which may be built into (e.g., integrated with) the computing device 1000. Illustrative I / O components 1014 include a microphone, mouse, keyboard, joystick, gamepad, game controller, satellite dish, scanner, printer, wireless device, etc. The I / O components 1014 can provide a natural user interface (NUI) that processes air gestures, speech, or other physiological inputs generated by a user. In some cases, the inputs can be transmitted to a suitable network element for further processing.A NUI can implement any combination of speech capture, stylus capture, face capture, biometric capture, gesture capture (both on-screen and off-screen), air gestures, head and eye tracking, and touch capture (as further described below) associated with a display of the Computing Device 1000. The Computing Device 1000 can include depth cameras, such as stereoscopic camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof, for gesture capture and recognition. Additionally, the Computing Device 1000 can include accelerometers or gyroscopes (e.g., as part of an inertial measurement unit (IMU)) that allow for motion detection. In some examples, the output from the accelerometers or gyroscopes can be used by the Computing Device 1000 to render immersive augmented reality or virtual reality.

[0243] The power supply 1016 can include a hardwired power supply, a battery power supply, or a combination thereof. The power supply 1016 can supply power to the computing device 1000 to allow the operation of the components of the computing device 1000.

[0244] The one or more presentation components 1018 can include a display (e.g., a monitor, a touchscreen, a television screen, a head-up display (HUD), other display types, or a combination thereof), speakers, and / or other presentation components. The one or more presentation components 1018 can receive data from other components (e.g., the one or more GPUs 1008, the one or more CPUs 1006, DPUs, etc.) and output the data (e.g., as an image, video, sound, etc.). EXEMPLARY DATA CENTER

[0245] Fig. Figure 11 illustrates an exemplary data center 1100 that can be used in at least one embodiment of the present disclosure. The data center 1100 can include an infrastructure layer 1110, a framework layer 1120, a software layer 1130, and / or an application layer 1140.

[0246] As in Fig. As shown in Figure 11, the infrastructure layer 1110 of the data center can contain a resource orchestrator 1112, grouped compute resources 1114 and node compute resources (“node RRs”) 1116(1)-1116(N), where “N” is any positive integer. In at least one embodiment, the node RRs 1116(1)-1116(N) may include, but are not limited to, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field programmable gate arrays (FPGAs), graphics processing units or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic solid-state memory), storage devices (e.g., solid-state or disk drives), network input / output (NW I / O) devices, network switches, virtual machines (VMs), power supply modules and / or cooling modules, etc.In some embodiments, one or more node RRs among node RRs 1116(1)-1116(N) may correspond to a server that has one or more of the compute resources mentioned above. Furthermore, in some embodiments, node RRs 1116(1)-1116(N) may contain one or more virtual components, such as vGPUs, vCPUs, and / or the like, and / or one or more of node RRs 1116(1)-1116(N) may correspond to a virtual machine (VM).

[0247] In at least one embodiment, the grouped compute resources 1114 can contain separate groupings of node RRs 1116, which are housed in one or more racks (not shown) or in many racks in data centers at different geographic locations (also not shown). Separate groupings of node RRs 1116 within grouped compute resources 1114 can contain grouped compute, network, memory, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, multiple node RRs 1116, including the CPUs, GPUs, DPUs, and / or other processors, can be grouped in one or more racks to provide compute resources to support one or more workloads.The one or more racks can also contain any number of power supply modules, cooling modules and / or network switches in any combination.

[0248] The resource orchestrator 1112 can configure or otherwise control one or more node RRs 1116(1)-1116(N) and / or grouped compute resources 1114. In at least one embodiment, the resource orchestrator 1112 can include an entity for managing the software design infrastructure (SDI) for the data center 1100. The resource orchestrator 1112 can include hardware, software, or a combination thereof.

[0249] In at least one embodiment, as in Fig. As shown in Figure 11, the framework layer 1120 can contain a job scheduler 1133, a configuration manager 1134, a resource manager 1136, and / or a distributed file system 1138. The framework layer 1120 can contain a framework that supports the software 1132 of the software layer 1130 and / or one or more applications 1142 of the application layer 1140. The software 1132 or the one or more applications 1142 can each contain web-based service software or applications such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. The framework layer 1120 can be a type of free and open-source software web application framework, such as Apache Spark™ (hereinafter "Spark"), which can use a distributed file system 1138 for processing large amounts of data (e.g., "Big Data"), without being limited to it.In at least one embodiment, the job scheduler 1133 can include a Spark driver to facilitate the scheduling of workloads supported by different layers of the data center 1100. The configuration manager 1134 can be able to configure different layers, such as the software layer 1130 and the framework layer 1120, which contains Spark and the distributed file system 1138, to support the processing of large amounts of data. The resource manager 1136 can be able to manage clustered or grouped compute resources allocated or assigned to support the distributed file system 1138 and the job scheduler 1133. In at least one embodiment, the clustered or grouped compute resources can include the grouped compute resource 1114 on the infrastructure layer 1110 of the data center.The resource manager 1136 can coordinate with the resource orchestrator 1112 to manage these allocated or assigned computing resources.

[0250] In at least one embodiment, the software 1130 contained in software layer 1132 may contain software used by at least sections of the node RRs 1116(1)-1116(N), the grouped compute resources 1114, and / or the distributed file system 1138 of framework layer 1120. One or more types of software may include, among others, web search software, email virus scanning software, database software, and streaming video content software.

[0251] In at least one embodiment, the applications 1142 contained in the application layer 1140 may include one or more types of applications used by at least sections of the node RRs 1116(1)-1116(N), the grouped compute resources 1114, and / or the distributed file system 1138 of the framework layer 1120. One or more types of applications may include, but are not limited to, any number of genome applications, cognitive computations, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments.

[0252] In at least one embodiment, the configuration manager 1134, the resource manager 1136, and / or the resource orchestrator 1112 can implement any number and type of self-modifying actions based on any set and type of data acquired in any technically feasible way. Self-modifying actions can relieve a data center operator of data center 1100 of potentially making poor configuration decisions and potentially avoiding underutilized and / or poorly functioning sections of a data center.

[0253] The Data Center 1100 may contain tools, services, software, or other resources to train one or more machine learning models or to predict or infer information using one or more machine learning models according to one or more embodiments described herein. For example, one or more machine learning models may be trained by calculating weighting parameters according to a neural network architecture, using software and / or computing resources described above in relation to the Data Center 1100.In at least one embodiment, trained or deployed machine learning models corresponding to one or more neural networks can be used to infer or predict information using the resources described above with reference to the Computing Center 1100 by using weighting parameters calculated by one or more training techniques such as, but not limited to, those described herein.

[0254] In at least one embodiment, the data center can use 1100 CPUs, application-specific integrated circuits (ASICs), GPUs, FPGAs, and / or other hardware (or equivalent virtual computing resources) to perform training and / or inference using the resources described above. Furthermore, one or more of the software and / or hardware resources described above can be configured as a service to allow users to train or infer information, such as image capture, speech capture, or other artificial intelligence services. EXEMPLARY NETWORK ENVIRONMENTS

[0255] Network environments suitable for implementing embodiments of the disclosure may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. The client devices, servers, and / or other device types (e.g., each device) may run on one or more instances of the one or more computing devices. Fig. 10. Implemented - for example, each device may contain similar components, features, and / or functionality to one or more computing devices 1000. If backend devices (e.g., servers, NAS, etc.) are implemented, the backend devices may also be included as part of a data center 1100, an example of which is given herein with reference to Fig. 11 is described in more detail.

[0256] The components of a network environment can communicate with each other over one or more networks, which can be wired, wireless, or both. The network can contain multiple networks or a network of networks. For example, the network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks such as the internet and / or a public switched telephone network (PSTN), and / or one or more private networks. If the network includes a wireless telecommunications network, components such as a base station, a communications tower, or even access points (as well as other components) can provide wireless connectivity.

[0257] Compatible network environments can contain one or more peer-to-peer network environments—in which case a server cannot be included in a network environment—and one or more client-server network environments—in which case one or more servers can be included in a network environment. In peer-to-peer network environments, the functionality described here can be implemented on any number of client devices with reference to one or more servers.

[0258] In at least one embodiment, a network environment can include one or more cloud-based network environments, a distributed computing environment, a combination thereof, etc. A cloud-based network environment can include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. A framework layer can include a framework to support software of a software layer and / or one or more applications of an application layer. The software or the one or more applications may each include web-based service software or applications. In embodiments, one or more of the client devices can use the web-based service software or applications (e.g.,by accessing the service software and / or applications via one or more application programming interfaces (APIs). The framework layer can be a type of free and open-source software web application framework that uses, for example, a distributed file system for processing large amounts of data (e.g., "Big Data"), but is not limited to this.

[0259] A cloud-based network environment can provide cloud computing and / or cloud storage, performing any combination (or one or more parts) of the computing and / or data storage functions described herein. Each of these different functions can be distributed across multiple locations of central or core servers (e.g., one or more data centers, which may be distributed across a state, region, country, the globe, etc.). If a connection to a user (e.g., a client device) is relatively close to one or more edge servers, one or more core servers can offload at least some functionality to the one or more edge servers. A cloud-based network environment can be private (e.g., restricted to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0260] The one or more client devices can incorporate at least some of the components, features, and functions of the one or more devices referred to here. Fig.The 10 exemplary computing devices described may include 1000. By way of example, and not as a limitation, a client device may be a personal computer (PC), a laptop, a mobile device, a smartphone, a tablet computer, a smartwatch, a portable computer, a personal digital assistant (PDA), an MP3 player, a virtual reality headset, a global positioning system (GPS) or global positioning device, a video player, a video camera, a surveillance device or surveillance system, a vehicle, a boat, a hydrofoil, a virtual machine, a drone, a robot, a handheld communication device, a hospital device, a gaming device or gaming system, an entertainment system, a vehicle computer system, an embedded system controller, a remote control, a device, a consumer electronics device, a workstation, an edge device,any combination of these described devices or any other suitable device may be embodied.

[0261] The disclosure can be described in the general context of computer code or machine-usable instructions, including computer-executable instructions such as program modules that are executed by a computer or other machine, such as a personal data assistant or other handheld device. Generally, program modules, which contain routines, programs, objects, components, data structures, etc., refer to code that performs specific tasks or implements certain abstract data types. The disclosure can be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc.The revelation can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are connected to each other via a network for communication.

[0262] As used herein, any mention of "and / or" in relation to two or more elements should be interpreted as referring to only one element or a combination of elements. For example, "element A, element B and / or element C" may include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. Furthermore, "at least one of element A or element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Additionally, "at least one of element A and element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0263] The subject matter of this disclosure is specifically described herein to satisfy legal requirements. However, the description itself is not intended to limit the scope of protection afforded by this disclosure. Rather, the inventors have considered that the claimed subject matter may also be embodied in other ways to include various steps or combinations of steps similar to those described in this document, in conjunction with other present or future technologies. Although the terms “step” and / or “block” may be used herein to denote various elements of the methods employed, these terms should not be interpreted as implying any particular sequence among or between the various steps disclosed herein, except where the sequence of each step is expressly described. EXEMPLARY LITERARY SUPPORT

[0264] One or more of the embodiments described below can exist alone or be combined in any way.

[0265] The disclosure of this application also contains the following numbered clauses: Clause 1 One or more processors comprising one or more processing units for: receiving user input specifying a type of parameterizable model; calculating one or more evaluation performance metrics corresponding to an implementation of the parameterizable model across a plurality of nodes of a distributed computing environment, based at least on the user input specifying the type of parameterizable model; and performing at least one of the validation or benchmarking of the parameterizable model, based at least on the calculation of the one or more evaluation performance metrics and the user input specifying the type of parameterizable model. Clause 2 The one or more processors according to Clause 1, wherein the user input further defines a sequence of actions that are associated with at least one of validation or benchmarking of the parameterizable model, wherein at least one action of the sequence of actions is a capture or recording task that is associated with at least one of validation or benchmarking, and wherein the one or more processing units further serve to: record or capture data according to the at least one action of the sequence of actions, based at least on the user input that defines the sequence of actions;and transferring the one or more evaluation performance metrics to a first compute node of the plurality of compute nodes, at least partially based on calculating the one or more evaluation performance metrics and recording or capturing the data according to at least one action. Clause 3 The one or more processors according to any of the preceding clauses, wherein the one or more processing units further serve to: transmit the one or more evaluation performance metrics to a first compute node of the plurality of compute nodes in response to at least one of the validation or benchmarking of the parameterizable model; and in response to the transmission of the one or more evaluation performance metrics to the first compute node of the plurality of compute nodes, receive, from the first compute node, an indication of whether a global validation has taken place, based at least on the first compute node with aggregated evaluation performance metrics from the plurality of compute nodes. Clause 4 The one or more processors according to Clause 3, wherein the first compute node aggregates evaluation performance metrics from the plurality of compute nodes by receiving one or more key performance indicators (KPIs) from at least two of the plurality of compute nodes and presenting at least one KPI of the one or more KPIs in a report. Clause 5 The one or more processors according to any of the preceding clauses, wherein the one or more evaluation performance metrics include one or more validation performance metrics and one or more benchmarking performance metrics, and wherein the one or more processing units further serve to: perform both validation and benchmarking of the parameterizable model. Clause 6 The one or more processors according to any of the preceding clauses, wherein the one or more processing units further serve to: in response to the validation or benchmarking of the parameterizable model, automatically initiate the presentation of a report on a user device, wherein the report contains at least one of the following: an initial visualization indicating changes in the evaluation performance metrics over time, sensor data associated with the parameterizable model, or a user instruction. Clause 7 The one or more processors according to any of the preceding clauses, wherein the one or more processors comprise at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulations; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system for performing real-time streaming; a system for generating or presenting one or more augmented reality content, virtual reality content, or mixed reality content; a system implemented using an edge device; a system implemented using a robot;a system for performing operations with conversational AI; a system for generating synthetic data; a system for implementation using one or more large language models (LLMs); a system for implementation using one or more vision language models (VLMs); a system that includes one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system that is at least partially implemented using cloud computing resources. Clause 8 System comprising one or more processing units for: receiving user input specifying at least one of the following: a type of parameterizable model or an identifier of one or more evaluation performance metrics; based on at least the user input, calculating the one or more evaluation performance metrics corresponding to an implementation of the parameterizable model across a plurality of compute nodes of a distributed computing environment; and based on at least the calculation of the one or more evaluation performance metrics and the user input, performing at least one of the validation or benchmarking of the parameterizable model. Clause 9 System according to Clause 8, wherein the user input further defines a sequence of actions that are associated with at least one of validation or benchmarking of the parameterizable model, wherein at least one action of the sequence of actions is a capture or recording task that is associated with at least one of validation or benchmarking, and wherein the one or more processing units further serve to: based at least on the user input that defines the sequence of actions, record or capture data according to the at least one action of the sequence of actions; and based at least partially on calculating the one or more evaluation performance metrics and recording or capturing the data according to the at least one action, transmit the one or more evaluation performance metrics to a first compute node of the plurality of compute nodes. Clause 10 System according to one of Clauses 8 or 9, wherein the one or more processing units further serve to: in response to at least one of the validation or benchmarking of the parameterizable model, transmit the one or more evaluation performance metrics to a first compute node of the plurality of compute nodes; and in response to the transmission of the one or more evaluation performance metrics to the first compute node of the plurality of compute nodes, receive from the first compute node an indication of whether a global validation, based at least on the first compute node with aggregated evaluation performance metrics from the plurality of compute nodes, has taken place. Clause 11 System according to Clause 10, wherein the first compute node aggregates evaluation performance metrics from the plurality of compute nodes by receiving one or more key performance indicators (KPIs) from at least two of the plurality of compute nodes and presenting at least one KPI of the one or more KPIs in a report. Clause 12 System according to one of Clauses 8 to 11, wherein the one or more evaluation performance metrics include one or more validation performance metrics and one or more benchmarking performance metrics, and wherein the one or more processing units further serve to: perform both validation and benchmarking of the parameterizable model. Clause 13 System according to any of Clauses 8 to 12, wherein the one or more processing units further serve to: in response to the validation or benchmarking of the parameterizable model, automatically initiate the presentation of a report on a user device, wherein the report contains at least one of the following: an initial visualization indicating changes in the evaluation performance metrics over time, sensor data associated with the parameterizable model, or a user instruction. Clause 14 System according to any of Clauses 8 to 13, wherein the system comprises at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulations; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system for performing real-time streaming; a system for generating or presenting one or more augmented reality content, virtual reality content, or mixed reality content; a system implemented using an edge device; a system implemented using a robot; a system for performing conversational AI operations;a system for generating synthetic data; a system for implementation using one or more large language models (LLMs); a system for implementation using one or more vision language models (VLMs); a system that includes one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system that is at least partially implemented using cloud computing resources. Clause 15 Procedure, comprising: based on at least one user input specifying a type of parameterizable model implemented across a plurality of compute nodes of a distributed computing environment, performing local validation and / or benchmarking by calculating one or more evaluation performance metrics; in response to performing local validation and / or benchmarking, transmitting the one or more evaluation performance metrics to a first compute node of the plurality of compute nodes; and in response to transmitting the one or more evaluation performance metrics to the first compute node of the plurality of compute nodes, causing the first compute node to aggregate one or more respective evaluation performance metrics from at least two compute nodes of the plurality of compute nodes. Clause 16 Procedure according to Clause 15, wherein the user input further defines a sequence of actions associated with at least one of validation or benchmarking of the parameterizable model, wherein at least one action of the sequence of actions is a capture or recording task associated with at least one of validation or benchmarking, and wherein the procedure further comprises: based at least on the user input defining the sequence of actions, recording or capturing data according to the at least one action of the sequence of actions; and at least partially based on calculating the one or more evaluation performance metrics and recording or capturing the data according to the at least one action, transmitting the one or more evaluation performance metrics to a first compute node of the plurality of compute nodes. Clause 17 Procedure according to one of Clauses 15 or 16, wherein the first compute node aggregates evaluation performance metrics from the plurality of compute nodes by receiving one or more key performance indicators (KPIs) from at least two of the plurality of compute nodes and presenting at least one KPI of the one or more KPIs in a report. Clause 18 Procedure according to one of Clauses 15 to 17, wherein the one or more evaluation performance metrics include one or more benchmarking performance metrics and wherein the procedure further comprises: performing benchmarking of the parameterizable model. Clause 19 Procedure according to any of Clauses 15 to 18, wherein the procedure further comprises: in response to the validation or benchmarking of the parameterizable model, automatically initiating the presentation of a report on a user device, wherein the report contains at least one of the following: an initial visualization indicating changes in the evaluation performance metrics over time, sensor data associated with the parameterizable model, or a user instruction. Clause 20: A procedure according to any of Clause 19, wherein the procedure is performed by at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulations; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system for performing real-time streaming; a system for generating or presenting one or more augmented reality content, virtual reality content, or mixed reality content; a system implemented using an edge device; a system implemented using a robot;a system for performing operations with conversational AI; a system for generating synthetic data; a system for implementation using one or more large language models (LLMs); a system for implementation using one or more vision language models (VLMs); a system that includes one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system that is at least partially implemented using cloud computing resources.

[0266] It is understood that the aspects and embodiments described above are purely exemplary and that modifications of details may be made within the scope of protection of the claims.

[0267] Each device, each method and each feature disclosed in the description, and (where applicable) the claims and drawings, may be provided independently or in any suitable combination.

[0268] Reference numerals appearing in the claims are for illustrative purposes only and do not restrict the scope of protection of the claims. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] US 16 / 101,232

[0166] Cited non-patent literature

[0000] Society of Automotive Engineers, SAE) (Standard No. J3016-201806, published on June 15, 2018, Standard No. J3016-201609, published on September 30, 2016

[0123]

Claims

[1] One or more processors comprising one or more processing units, for: Receiving user input that specifies a type of parameterizable model; Calculating one or more performance metrics corresponding to an implementation of the parameterizable model across a multitude of nodes in a distributed computing environment, based at least on user input specifying the type of parameterizable model; and Performing at least one validation or benchmarking of the parameterizable model, based at least on calculating one or more evaluation performance metrics and user input that specifies the type of parameterizable model. [2] The one or more processors according to claim 1, wherein the user input further defines a sequence of actions that are assigned to at least one of validation or benchmarking of the parameterizable model, wherein at least one action of the sequence of actions represents a capture or recording task that is assigned to at least one of validation or benchmarking, and wherein the one or more processing units further serve to: Recording or capturing data according to at least one action of the sequence of actions, based at least on user input that defines the sequence of actions; and Transferring one or more evaluation performance metrics to a first compute node of the plurality of compute nodes, at least partially based on calculating one or more evaluation performance metrics and recording or capturing the data according to at least one action. [3] The one or more processors according to any of the preceding claims, wherein the one or more processing units further serve to: Transferring one or more evaluation performance metrics to a first compute node of the multitude of compute nodes in response to at least one of the validation or benchmarking of the parameterizable model; and in response to the transmission of one or more evaluation performance metrics to the first compute node of the multitude of compute nodes, receiving from the first compute node an indication of whether a global validation has taken place, based at least on the first compute node with aggregated evaluation performance metrics from the multitude of compute nodes. [4] The one or more processors according to claim 3, wherein the first compute node aggregates evaluation performance metrics from the plurality of compute nodes by receiving one or more key performance indicators (KPIs) from at least two of the plurality of compute nodes and presenting at least one KPI of the one or more KPIs in a report. [5] The one or more processors according to any of the preceding claims, wherein the one or more evaluation performance metrics include one or more validation performance metrics and one or more benchmarking performance metrics, and wherein the one or more processing units further serve to: Performing both validation and benchmarking of the parameterizable model. [6] The one or more processors according to any of the preceding claims, wherein the one or more processing units further serve to: In response to the validation or benchmarking of the parameterizable model, automatically initiating the presentation of a report on a user device, wherein the report contains at least one of the following: an initial visualization indicating changes in the evaluation performance metrics over time, sensor data associated with the parameterizable model, or a user instruction. [7] The one or more processors according to any one of the preceding claims, wherein the one or more processors comprise at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulations; a system for conducting collaborative content creation for 3D assets; a system for performing deep learning operations; a system for performing real-time streaming; a system for generating or presenting one or more augmented reality content, virtual reality content, or mixed reality content; a system that is implemented using an edge device; a system that is implemented using a robot; a system for performing operations using conversational AI; a system for generating synthetic data; a system for implementation using one or more large language models (LLMs); a system for implementation using one or more Vision Language Models (VLMs); a system that includes one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system that is implemented at least partially using cloud computing resources. [8] System comprising one or more processing units, for: Receiving user input that specifies at least one of the following: a type of parameterizable model or an identifier of one or more evaluation performance metrics; based on at least the user input, calculating one or more evaluation performance metrics corresponding to an implementation of the parameterizable model across a multitude of compute nodes in a distributed computing environment; and based at least on calculating one or more evaluation performance metrics and user input, performing at least one validation or benchmarking of the parameterizable model. [9] System according to claim 8, wherein the user input further defines a sequence of actions that are assigned to at least one of validation or benchmarking of the parameterizable model, wherein at least one action of the sequence of actions represents a capture or recording task that is assigned to at least one of validation or benchmarking, and wherein the one or more processing units further serve to: based at least on user input that defines the sequence of actions, recording or capturing data according to at least one action of the sequence of actions; and at least partially based on calculating one or more evaluation performance metrics and recording or capturing the data according to at least one action, transferring the one or more evaluation performance metrics to a first computation node of the plurality of computation nodes. [10] System according to one of claims 8 or 9, wherein the one or more processing units further serve to: as a response to at least one of the validation or benchmarking results of the parameterizable model, transferring one or more evaluation performance metrics to a first compute node of the multitude of compute nodes; and in response to the transmission of one or more evaluation performance metrics to the first compute node of the multitude of compute nodes, receiving from the first compute node an indication of whether a global validation, based at least on the first compute node with aggregated evaluation performance metrics from the multitude of compute nodes, has taken place. [11] System according to claim 10, wherein the first compute node aggregates evaluation performance metrics from the plurality of compute nodes by receiving one or more key performance indicators (KPIs) from at least two of the plurality of compute nodes and presenting at least one KPI of the one or more KPIs in a report. [12] System according to any one of claims 8 to 11, wherein the one or more evaluation performance metrics include one or more validation performance metrics and one or more benchmarking performance metrics, and wherein the one or more processing units further serve to: Performing both validation and benchmarking of the parameterizable model. [13] System according to any one of claims 8 to 12, wherein the one or more processing units further serve to: In response to the validation or benchmarking of the parameterizable model, automatically initiating the presentation of a report on a user device, wherein the report contains at least one of the following: an initial visualization indicating changes in the evaluation performance metrics over time, sensor data associated with the parameterizable model, or a user instruction. [14] System according to any one of claims 8 to 13, wherein the system comprises at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulations; a system for conducting collaborative content creation for 3D assets; a system for performing deep learning operations; a system for performing real-time streaming; a system for generating or presenting one or more augmented reality content, virtual reality content, or mixed reality content; a system that is implemented using an edge device; a system that is implemented using a robot; a system for performing operations using conversational AI; a system for generating synthetic data; a system for implementation using one or more large language models (LLMs); a system for implementation using one or more Vision Language Models (VLMs); a system that includes one or more virtual machines (VMs); a system that is at least partially implemented in a data center or a system that is implemented at least partially using cloud computing resources. [15] Procedure, encompassing: based on at least one user input specifying a type of parameterizable model implemented across a multitude of compute nodes in a distributed computing environment, performing local validation and / or benchmarking by calculating one or more evaluation performance metrics; in response to performing local validation and / or benchmarking, transferring one or more evaluation performance metrics to a first compute node of the multitude of compute nodes; and In response to the transfer of one or more evaluation performance metrics to the first compute node of the plurality of compute nodes, cause the first compute node to aggregate one or more respective evaluation performance metrics from at least two compute nodes of the plurality of compute nodes. [16] Method according to claim 15, wherein the user input further defines a sequence of actions that are associated with at least one of validation or benchmarking of the parameterizable model, wherein at least one action of the sequence of actions represents a capture or recording task that is associated with at least one of validation or benchmarking, and wherein the method further comprises: based at least on user input that defines the sequence of actions, recording or capturing data according to at least one action of the sequence of actions; and at least partially based on calculating one or more evaluation performance metrics and recording or capturing the data according to at least one action, transferring the one or more evaluation performance metrics to a first computation node of the plurality of computation nodes. [17] Method according to one of claims 15 or 16, wherein the first computation node aggregates evaluation performance metrics from the plurality of computation nodes by receiving one or more key performance indicators (KPIs) from at least two of the plurality of computation nodes and presenting at least one KPI of the one or more KPIs in a report. [18] Method according to any one of claims 15 to 17, wherein the one or more evaluation performance metrics include one or more benchmarking performance metrics and wherein the method further comprises: Performing benchmarking of the parameterizable model. [19] Method according to any one of claims 15 to 18, wherein the method further comprises: In response to the validation or benchmarking of the parameterizable model, automatically initiating the presentation of a report on a user device, wherein the report contains at least one of the following: an initial visualization indicating changes in the evaluation performance metrics over time, sensor data associated with the parameterizable model, or a user instruction. [20] The method of claim 19, wherein the method is carried out by at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulations; a system for conducting collaborative content creation for 3D assets; a system for performing deep learning operations; a system for performing real-time streaming; a system for generating or presenting one or more augmented reality content, virtual reality content, or mixed reality content; a system implemented using an edge device; a system that is implemented using a robot; a system for performing operations with conversational AI; a system for generating synthetic data; a system for implementation using one or more large language models (LLMs); a system for implementation using one or more Vision Language Models (VLMs); a system that includes one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system that is implemented at least partially using cloud computing resources.

Images