Dataset generation and expansion for machine learning models

Abstract

To provide attribute-controllable generation of a data set for a machine learning model.SOLUTION: A machine learning model (MLM) may be trained and evaluated. Attribute-based performance metrics may be analyzed to identify attributes for which the MLM is performing below a threshold when each are present in a sample. A generative neural network (GNN) may be used to generate samples including compositions of the attributes, and the samples may be used to augment data used to train the MLM. This may be repeated until one or more criteria are satisfied. In various examples, a temporal sequence of data items, such as frames of a video may be formed which may form samples of the data set. Sets of attribute values may be determined based on one or more temporal scenarios to be represented in the data set, and one or more GNNs may be used to generate the sequence to depict information corresponding to the attribute values.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to providing data set generation and augmentation for machine learning models.

Background Art

[0002] The performance of a machine learning model (MLM), such as a deep neural network (DNN), can be improved by modifying the architecture of the MLM or the data used to train the MLM. Existing solutions attempt to improve the data used to train the MLM by collecting more data. However, collecting real-world data is a laborious, costly, and time-consuming task that requires countless human and computational resources. Even if large amounts of data can be collected, certain scenarios that should be captured to produce a robust and generalized model may be rare and constitute a low percentage of the data. Thus, the MLM may still not function well enough in these few scenarios.

[0003] Data augmentation techniques can be employed to reduce the amount of real-world data that needs to be collected to train a Multi-Level Model (MLM). Existing data augmentation techniques include rotating, flipping, cropping images, or otherwise modifying existing data. These techniques can improve MLM accuracy when these modified forms are observed by the trained MLM. However, these techniques may not address accuracy problems associated with scenarios that exist in small numbers. For example, if the data in the training dataset (e.g., faces) is not sufficiently varied, or if certain characteristics are over- or under-presented in the training dataset, the face detection network may have lower accuracy. Thus, while more data can be acquired using conventional data acquisition and augmentation techniques, existing solutions cannot explain how much data of a particular type is needed to obtain a robust and generalized model. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] U.S. Patent Application No. 16 / 101,232 [Overview of the project]

[0005] Embodiments of this disclosure relate to attribute-controllable generation of datasets for machine learning models. In embodiments, a generative neural network (GNN) may be used to generate at least one class of samples having one or more attributes based on at least the distribution of attributes of training, validation, and / or test data.

[0006] In contrast to conventional methods such as those described above, the disclosed methods provide techniques for determining what types of data and how much of a particular type of data is required for training, validating, and / or testing an MLM. In embodiments of the Disclosure, the MLM may be trained and evaluated, and one or more attribute-based performance metrics may be analyzed to identify one or more attributes and / or combinations of attributes in which the MLM is performing below absolute and / or relative performance thresholds. A GNN may be used to generate additional training data having one or more samples, including a synthesis of multiple attributes, and the newly generated training data may be used to augment the existing data in the dataset used to train the MLM. In at least one embodiment, this process may be repeated until one or more criteria are met. Aspects of the Disclosure also realize generating a dataset for an MLM based on at least certain attributes (e.g., a list of attributes, an indicator of attributes, etc.) associated with at least one class that will be represented in the dataset. The Disclosure further realizes generating a time-series sequence of data items, such as frames of a video, which may form one or more samples of the dataset. A set of attribute values ​​may be determined based on at least one or more time-series scenarios that will be represented in the data set, and one or more GNNs may be used to generate sequences for describing the information corresponding to the attribute values.

[0007] The system and method for attribute-controllable generation of datasets for machine learning models are described in detail below with reference to the attached diagrams. [Brief explanation of the drawing]

[0008] [Figure 1] This is an illustration of an exemplary process that may be used to evaluate one or more machine learning models using attribute-controllable generation of one or more datasets, according to some embodiments of the present disclosure. [Figure 2] This is an illustration of exemplary processes that may be performed to controllly generate data for one or more machine learning models based on one or more attributes, according to some embodiments of the present disclosure. [Figure 3] This flowchart illustrates a method for controllingly generating one or more samples based on at least evaluating one or more performance metrics of one or more attributes of one or more machine learning models, according to some embodiments of the present disclosure. [Figure 4] This flowchart illustrates a method, according to some embodiments of the present disclosure, for controllably generating one or more samples of one or more attributes using one or more performance metric values. [Figure 5] This is an illustration of exemplary processes that may be performed according to some embodiments of the present disclosure to controllly generate data for one or more machine learning models based on time-dependent patterns of one or more attributes. [Figure 6] This is an illustration of an example of a frame that may be generated to capture at least a portion of a time-series scenario according to some embodiments of the present disclosure. [Figure 7] This is an illustration used to illustrate examples of how one or more time-series patterns may be extracted from a reference data set, according to some embodiments of the present disclosure. [Figure 8] This flowchart illustrates a method for controllingly generating one or more samples based on at least associating one or more time-series patterns with one or more time-series scenarios, according to some embodiments of the present disclosure. [Figure 9] This is an illustration of an exemplary process that may be used for attribute-controllable generation of one or more data sets according to some embodiments of the present disclosure. [Figure 10] This flowchart illustrates a method for controllingly generating one or more samples based on at least an analysis of input data, according to some embodiments of the present disclosure. [Figure 11A] This is an illustration of an exemplary autonomous vehicle according to some embodiments of the present disclosure. [Figure 11B] Figure 11A shows examples of camera positions and fields of view of an exemplary autonomous vehicle according to some embodiments of the present disclosure. [Figure 11C] Figure 11A is a block diagram of an exemplary system architecture of an exemplary autonomous vehicle according to some embodiments of the present disclosure. [Figure 11D] This is a system diagram of communication between a cloud-based server and the exemplary autonomous vehicle shown in Figure 11A, according to some embodiments of the present disclosure. [Figure 12] This is a block diagram of an exemplary computing device suitable for use in implementing some embodiments of the present disclosure. [Figure 13] This is an exemplary data center block diagram suitable for use in implementations of some embodiments of the present disclosure. [Modes for carrying out the invention]

[0009] Systems and methods are disclosed for attribute-controllable generation of data for training, validation, and testing machine learning models. While this disclosure may be described in relation to an exemplary autonomous vehicle 1100 (or also referred to herein as “Vehicle 1100” or “Ego Vehicle 1100,” examples thereof are described with respect to Figures 11A–11D), this is not intended to be limiting. For example, the systems and methods described herein may be used by, but are not limited to, non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more adaptive driver assistance systems (ADAS)), manned and unmanned robots or robotics platforms, warehouse vehicles, off-road vehicles, vehicles attached to one or more trailers, flying vessels, boats, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, submarines, drones, and / or other vehicle types. In addition, this disclosure may describe, but is not intended to limit, the systems and methods described herein may be used in any other technological space where augmented reality, virtual reality, mixed reality, robotics, security and surveillance, simulation and digital twins, autonomous or semi-autonomous machine applications, and / or machine learning models may be used.

[0010] In some embodiments, a generative neural network (GNN), such as a generative adversarial network (GAN), may be used to generate at least one class of samples having one or more attributes based on at least the distribution of attributes in the training, validation, and / or test data.

[0011] In contrast to conventional methods such as those described above, the disclosed methods provide techniques for determining what types of data and how much of a particular type of data is required to train, validate, and / or test an MLM. In at least one embodiment, an MLM may be trained and evaluated using a training data set. Rather than evaluating the overall performance of an MLM, the performance of an MLM may be evaluated using one or more key performance indicators (KPIs) that quantify the MLM performance with respect to specific attributes and / or combinations of attributes. Attribute-based KPIs may be analyzed to identify one or more attributes and / or combinations of attributes that the MLM is performing below absolute and / or relative performance thresholds (e.g., inference accuracy thresholds). A GNN may be used to generate additional training data corresponding to the identified attributes. For example, a GNN may generate one or more samples, each sample may contain a synthesis of multiple attributes. Training data may be used to expand the training data set when training an MLM. In at least one embodiment, this process may be repeated until one or more criteria are met, such as the KPI indicating that the MLM is performing above a performance threshold for each attribute and / or combination of attributes.

[0012] As a non-limiting example, one or more attributes may relate to one or more parts of a person depicted in one or more images, for example, the person's age, the person's hair length, the person's head position, whether the person is wearing glasses, whether the person has a beard, the person's emotions, the person's blink rate, the person's eyelid opening, the person's eye makeup, the amplitude of the person's blink, the duration of the person's blink, the person's facial pattern, whether the person is wearing a mask, the lighting conditions of the person, or one or more attributes that define the person's facial expression. Other examples of attributes include object background and / or foreground emphasis (e.g., visual emphasis and / or focus).

[0013] The content and / or parts of people and / or other objects depicted in one or more images may vary according to the application. For example, for training an MLM for face recognition, face detection, or face landmark detection, an entire face image that may be generated by (or cropped from a larger image generated by) a GNN may be used. For training an MLM for eye-based drowsiness detection or eye opening / closing detection, an eye crop region may be generated by (or cropped from a larger image generated by) a GNN. The outputs of these MLMs can be useful for controlling one or more operations of a vehicle, such as vehicle 1100, (e.g., driver attention monitoring, passenger profile / account management, etc.), but have a much broader adaptability.

[0014] Aspects of the present disclosure may also enable generating a data set for an MLM, regardless of whether the MLM has been trained (e.g., beyond augmenting the training data set used to train the MLM). For example, at least some of the initial training data set may be generated based at least in part on specific attributes related to at least one class to be represented in the data set. Additionally or alternatively, at least some of the validation and / or test data set may be generated based at least in part on specific attributes related to at least one class to be represented in the data set.

[0015] In one or more embodiments, the GNN can be used to generate one or more portions of a training, validation, and / or test data set based at least on a set of attributes. For example, the GNN can be used to generate a training data set according to a desired distribution of attributes and / or combinations of attributes of a training, validation, or test data set and / or the overall data set from which one or more of those data sets are formed. One example of such a distribution includes a uniform distribution of attributes and / or combinations of attributes. Using such techniques, a data set can be customized for a particular scenario related to the attributes.

[0016] The disclosed techniques can be used to define and / or change the distribution of attributes and / or combinations of attributes in a data set used for training, validation, and / or testing. By way of example and not limitation, at least 80% of the data can be reserved for training (e.g., 95%), and any remaining data can be used for validation and / or testing (e.g., 5%). The validation and / or test data set can be augmented using the GNN to include one or more additional samples corresponding to rare attributes and / or combinations thereof so as to include sufficient test data for those scenarios.

[0017] This disclosure further enables the generation of a time-series sequence of data items, such as video frames, which can form one or more samples of a training, validation, and / or test data set. In at least one embodiment, a frame (or more generally, a sample) in a sequence of frames may be an assigned set of one or more attribute values ​​that will be depicted (or more generally represented or performed) within the frame. The set of attribute values ​​may be determined based on at least one or more time-series scenarios that will be represented using one or more attribute values. For example, a time-series scenario may be mapped to the attribute values ​​of a frame, and one or more GNNs may be used to generate the frames and depict the visual information corresponding to the attribute values. Non-limiting examples of time-series scenarios include scenarios involving one or more blinking patterns (e.g., blinking frequency, amplitude, speed, duration), one or more facial patterns (e.g., yawning, not yawning), and / or one or more head positions (e.g., dozing, alert).

[0018] In at least one embodiment, a time-series scenario may correspond to one or more ground truth inferences that will be made by the MLM based on the generated samples. For example, if the MLM is to infer sleepiness levels, these sleepiness levels may be mapped to one or more sets of attribute values. Various methods may be used to determine time-series patterns of one or more attributes based on one or more ground truth inferences. For example, time-series patterns of attributes and / or combinations of attributes may be determined at least by analyzing and / or identifying corresponding time-series patterns in the real world or observational data corresponding to one or more ground truth inferences.

[0019] The systems and methods described herein may be used by, but are not limited to, non-autonomous vehicles, semi-autonomous vehicles (for example, in one or more adaptive driver assistance systems (ADAS)), manned and unmanned robots or robotics platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, airships, boats, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, submarines, drones, and / or other vehicle types. Furthermore, the systems and methods described herein may be used, not limited to but exemplify, for a variety of purposes, including, but are examples, for machine control, machine movement, machine operation, synthetic data generation, model training, cognition, augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital twins, data center processing, conversational AI, optical transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation of 3D assets, cloud computing and / or any other suitable applications.

[0020] The disclosed embodiments may include a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, cognitive systems for autonomous or semi-autonomous machines), systems implemented using robots, aerospace systems, medical systems, marine systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems incorporating one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems at least partially implemented in a data center, systems for performing conversational AI operations, systems for performing optical transport simulations, systems for performing collaborative content creation of 3D assets, systems implemented at least partially using cloud computing resources, and / or other types of systems.

[0021] Referring to Figure 1, Figure 1 illustrates an exemplary process 100 that may be used to evaluate one or more machine learning models using attribute-controllable generation of one or more datasets, according to some embodiments of the present disclosure. Various components are shown in Figure 1 and other figures of this specification. It should be understood that this and other configurations described herein are provided for illustrative purposes only. Other configurations and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those illustrated, and some elements may be excluded together. Furthermore, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in combination with other components, and in any appropriate combination and location. Various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory. In some embodiments, the systems, methods, and processes described herein may be implemented using components, features, and / or functionalities similar to those of the exemplary autonomous vehicle 1100 in Figures 11A–11D, the exemplary computing device 1200 in Figure 12, and / or the exemplary data center 1300 in Figure 13.

[0022] Figure 1 shows the components of the system, which may include one or more machine learning models (MLMs) 104, an output analyzer 108, a dataset classifier 112, one or more generating MLMs 116, and an MLM trainer 120.

[0023] At a high level, process 100 may include MLM 104 receiving one or more inputs, for example, one or more samples from a dataset 122 (e.g., a training dataset), and generating one or more outputs, for example, output data 124 (e.g., tensor data) from one or more inputs. As shown in Figure 1, dataset 122 may be applied to MLM 104 by an MLM trainer 120. However, dataset 122 may be applied to MLM 104 by different MLM trainers. Process 100 may also include output analyzer 108 receiving one or more inputs, for example, output data 124, and generating one or more outputs, for example, performance data 128 (e.g., representing one or more performance metrics of at least one attribute) from one or more inputs. The data set determination unit 112 may receive one or more inputs, for example, performance data 128, and generate one or more outputs, for example, control data 130 from one or more inputs (for example, at least one input corresponding to at least one value of at least one attribute). The generating MLM 116 may receive one or more inputs, for example, control data 130, and generate one or more outputs, for example, a generated data set 132 from one or more inputs. Process 100 may repeat any number of iterations. For subsequent iterations, the MLM trainer 120 may train and / or refine the MLM 104 by applying at least a portion of the generated data set 132 from one or more previous iterations to the MLM 104. In at least one embodiment, the data set determination unit 112 may determine in any instance of process 100 that the performance of the MLM 104 is sufficient and / or that process 100 should be terminated in any other way without generating the generated data set 132. MLM104 may be deployed for and / or undergo additional verification, testing, and / or adaptation, at least based on its determination.

[0024] MLM104 and other MLMs described herein may include any type of machine learning model, e.g., machine learning models using linear regression, logistic regression, decision trees, support vector machines (SVMs), naive Bayes, k-nearest neighbors (Knn), K-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoders, convolutions, recurrences, perceptrons, long / short-term memory (LSTMs), Hopfield, Boltzmann, deep beliefs, deconvolution, adversarial generation, liquid state machines, etc.), and / or other types of machine learning models. In various examples, an MLM may include one or more convolutional neural networks.

[0025] As an example, and not an limitation, MLM104 may include one or more face detection networks, face recognition networks, facial landmark detection networks, eye open / close classification networks, emotion classification networks, and / or drowsiness detection networks. For example, MLM104 may be trained using MLM trainer 120 at least in part for face recognition, face detection, facial landmark detection, eye-based drowsiness detection, and / or eye open / close detection or classification.

[0026] Data set 122 may include training, validation, or test data. For example, data set 122 may be used by MLM trainer 120 to train MLM 104, to validate MLM 104, or to test MLM 104. Similarly, data set 132 generated from one or more iterations of process 100 may be used by MLM trainer 120 to train MLM 104, to validate MLM 104, or to test MLM 104 and / or one or more other MLMs. The generated data set 132 may be applied in subsequent iterations of the process, and more generally, one or more samples from the generated data set 132 may be applied to MLM 104, and / or used to generate one or more samples applied to MLM 104. In at least one embodiment, one or more samples from the generated data set 132 may be used by the MLM trainer 120 to extend one or more samples from data set 122 and / or previously used to train the MLM 104. For example, the MLM trainer 120 can retrain the MLM 104 (or train a different MLM) using an extended data set that includes samples from the generated data set 132 and samples from previously used to train the MLM 104, or it can use samples from the generated data set 132 to refine the trained MLM 104.

[0027] Such applications of the generated data set 132 may occur periodically, sequentially, sequentially, or with any appropriate regularity over iterations of process 100. In one or more embodiments, data set 122 may be applied to MLM 104 over several iterations of process 100 to multiple instances of the generated data set 132, which may be selected from MLM 104 and / or one or more other MLMs for training, validation, and / or testing, and / or combined for application thereto.

[0028] In one or more embodiments, the data set 122 and / or the generated data set 132 may define one or more samples applied to the MLM 104 by the MLM trainer 120 in process 100. A sample may correspond to at least one class (e.g., an output class of the MLM 104) having one or more attributes. Examples of attributes of a machine learning model used to generate inferences about a face image (e.g., a photograph or rendering) may include, but are not limited to, age, ethnicity, long / short hair, hair length, eyes open / closed, degree of eye opening, presence / absence of glasses, presence / absence of a beard, orientation or pose of the head (or other body part) or body, lighting conditions (e.g., shaded, bright, lighting color, light level, etc.), emotion (e.g., happy, sad, neutral, etc.), presence and / or type of eye makeup, and / or sleepiness level.

[0029] The output analyzer 108 may be configured to generate performance data 128 using output data 124. The output data 124 may represent one or more outputs from the MLM 104. In at least one embodiment, the output data 124 may include at least a portion of the tensor data from the MLM 104. The output analyzer 108 may generate performance data 128 based at least on analyzing the output data 124. The analysis of the output data 124 may be performed using a variety of techniques. In at least one embodiment, the output analyzer 108 may post-process at least some of the output data 124 to determine, for example, one or more inferred or predicted outputs of the MLM 104 (e.g., one or more outputs that the MLM 104 is trained to infer or is being trained to infer). The output analyzer 108 may analyze the post-processed data to determine the performance data 128. Additionally or alternatively, the output analyzer 108 may analyze one or more portions of the output data 124 using one or more MLMs trained to predict performance data 128 based at least partially on the output data 124.

[0030] In at least one embodiment, the output analyzer 108 can determine and / or generate one or more performance metrics represented by at least a portion of the performance data 128, based at least on an analysis of the output data 124. The performance metrics determined using the output analyzer 108 may be for one or more attributes of the samples applied to the MLM 104 to generate the output data 124, or may correspond to at least one of the samples. For example, instead of evaluating the overall performance of the MLM 104, the output analyzer 108 can evaluate the performance of the MLM 104 using one or more key performance indicators (KPIs) that quantify and / or correspond to the performance of the MLM 104 in terms of one or more values ​​of specific attributes and / or combinations of attributes. For example, for a face detection network, the KPI could correspond to how accurately MLM104 predicted the presence of a face when the input sample depicted a person without a mask versus a person with a mask (e.g., a first attribute with a value indicating presence or absence) and / or a person with a short beard versus a person with a long beard (e.g., a second attribute with a value indicating the length of one or more beards).

[0031] In at least one embodiment, the output analyzer 108 can evaluate the sample using the sample's ground truth prediction to determine one or more performance metrics indicating whether the MLM 104 made accurate or inaccurate predictions for one or more corresponding samples. In at least one embodiment, the output analyzer 108 can evaluate the sample using the sample's ground truth prediction to determine one or more performance metrics indicating how close the prediction made by the MLM 104 is to the ground truth prediction of one or more corresponding samples. One or more performance metrics may correspond to one or more attributes, as described herein. For example, when one or more values ​​of one or more attributes are determined to be performed by one or more samples corresponding to a performance metric, the performance metric of one or more attributes can describe performance.

[0032] In at least one embodiment, the output analyzer 108 may use attribute labels for one or more samples, indicating that one or more samples represent one or more attributes and / or their values. For example, each sample may be labeled with one or more attributes and / or attribute values ​​that are performed by or determined to be associated with the sample. Using samples with attribute labels corresponding to attributes, the output analyzer 108 can calculate performance metrics that are at least partially based on those attributes. One or more of the attribute labels may be assigned to one or more samples using human labeling and / or machine labeling (for example, using one or more MLMs to predict one or more attributes and / or their values).

[0033] Various types of performance metrics may be used. Non-restrictive examples include those based on mean squared error (MSE), normalized MSE, square root MSE, R squared, true positive rate, false positive rate, F score or F value, precision, precision, recall, intersection or Jackard coefficient for union, mean percentage error, error rate, etc. As a simple example, a 50% performance metric value for an attribute may be based on the output analyzer 108 determining that the MLM 104 provided 50% accurate predictions for time when the sample corresponded to an attribute (e.g., had a corresponding attribute label).

[0034] The data set determination unit 112 can analyze the performance data 128 to generate control data 130 for the generated MLM 116. For example, the data set determination unit 112 can generate control data 130 based on at least one or more performance metrics (e.g., attribute-based KPIs) determined using the output analyzer 108. The data set determination unit 112 can determine one or more characteristics of the generated data set 132 based on at least the analysis of the performance data 128 and provide control data 130 corresponding to one or more characteristics to construct the generated data set 132.

[0035] In at least one embodiment, the data set determination 112 analyzes one or more performance metrics to identify one or more attributes and / or combinations of attributes and / or attribute values ​​in which the MLM 104 is performing below absolute and / or relative performance thresholds. For example, an attribute (and / or attribute value) may be identified or selected by the data set determination 112 based at least on the performance metric value of the attribute (and / or attribute value) that indicates lower performance of the attribute (and / or attribute value) compared to the performance metric value of at least one other attribute. In one example, an attribute or attribute value is selected based at least on being one of the three worst performing attributes and / or corresponding to performance below a threshold calculated based on the performance of multiple attributes and / or attribute values. In addition or by alternative means, attributes (and / or attribute values) may be identified or selected by the data set decisioner 112 based at least on the performance metric value of an attribute (and / or attribute value) that indicates lower performance of the attribute (and / or attribute value) against a threshold, which does not necessarily have to be specific to the attribute (and / or attribute value) being considered. In one example, any attribute (and / or attribute value) or combination of attributes (and / or attribute values) is selected based at least on the corresponding inference accuracy being less than 65%.

[0036] In addition or by alternative means, the data set determination unit 112 may include one or more MLMs trained to determine one or more characteristics and / or generate control data 130 based on at least one or more portions of the performance of the performance data 128 and / or output data 124, for example, one or more performance metrics and / or one or more portions of the data generated therefrom as input to the output data 124 or an MLM.

[0037] One example of one or more characteristics is the inclusion of a large number of samples in the generated data set 132 to include one or more attributes and / or combinations of attributes (and / or attribute values). For example, the quantity of samples for an attribute (and / or attribute value) may be based at least on the value of the performance metric for that attribute (and / or attribute value). As an example and not an limitation, the quantity may increase with the distance of the value from a threshold, e.g., a threshold used to identify or select an attribute (and / or attribute value), or otherwise be based at least on the distance of the value from the threshold. Additionally or alternatively, one or more characteristics may include the assignment of samples to a distribution of samples generated using the generated MLM 116. For example, a fixed or calculated number of samples to be generated using the generated MLM 116 may be assigned among attributes, attribute values, and / or combinations of attributes (e.g., at least on the corresponding values ​​of one or more performance metrics). As an example, and not an limitation, the assignment of an attribute (and / or attribute value) may increase with the performance metric value of the attribute (and / or attribute value) for the value of one or more other attributes (and / or attribute values), or otherwise may be based at least on the performance metric value. Attributes as described herein may also refer to a single attribute or a combination of attributes (composite attribute). Furthermore, attribute values ​​as described herein may refer to a single attribute value or a combination of attribute values.

[0038] In various examples, the dataset classifier 112 may analyze performance data 128 (e.g., one or more performance metrics) across attributes to identify underperformance of composite attributes formed by combinations of attributes. Doing so may allow for the identification of underperformance of edge cases or corner cases that may not be adequately represented in the data used to train the MLM 104. For example, the MLM 104 may function well overall when a child is in the image input to the MLM 104, and may function well overall when a person wearing glasses is in the image input to the MLM 104. However, the MLM 104 may not function well when a child wearing glasses is in the image input to the MLM 104. As an example, here “child” may refer to a specific range of age values ​​identified by the dataset classifier 112 based on an analysis of performance with respect to the age attribute. As another example, “child” may refer to the binary value of an attribute that indicates whether a child is present and is identified by the dataset classifier 112. This type of synthetic training data may have been present in only small quantities in the data used to train MLM104. By analyzing the performance data 128 across attributes, the dataset classifier 112 may determine that it is generating one or more additional samples of these cases.

[0039] As described herein, in at least one embodiment, the data set determination 112 may determine in any instance of process 100 that the performance of MLM 104 is sufficient and / or otherwise process 100 and / or its iterations may terminate at any point (for example, without generating the generated data set 132). For example, analysis of performance data 128 may result in the data set determination 112 not selecting any attributes and / or combinations of attributes of the generated data set 132, the MLM 104 may reach a threshold level of performance, process 100 may be executed a threshold number of iterations, and / or a threshold number of samples may be generated.

[0040] In one or more embodiments, the control data 130 of the generating MLM 116 may represent at least one input to the generating MLM 116 corresponding to at least one attribute, causing the generating MLM 116 to generate one or more samples that implement at least one attribute having at least one value. Using an example of a child wearing glasses, at least one input may cause the generating MLM 116 to generate one or more images depicting a child wearing glasses. If at least one value corresponds to a range of values, the control data 130 may cause the generating MLM 116 to generate samples that are randomly sampled, indicate a range, or are selected from a range in other ways. In one or more embodiments, the generating MLM 116 may perform synthetic generation of attributes in latent space (e.g., synthesis of distributions).

[0041] While any suitable generative MLM and process may be used to generate the generated data set 132, Figure 2 provides an example of a suitable technique. Referring here to Figure 2, Figure 2 illustrates an exemplary process 200 that may be used to controllly generate data for one or more machine learning models based on one or more attributes, according to some embodiments of the present disclosure. Figure 2 shows the components of a system, which may include an unconditional generator 202, a controllable generator 204, and an attribute classifier 206. In one or more embodiments, the unconditional generator 202, the controllable generator 204, and the attribute classifier 206 may form at least a part of the generative MLM 116 of Figure 1.

[0042] At a high level, process 200 may include an unconditional generator receiving one or more inputs, e.g., at least a portion of control data 130 (e.g., one or more inputs used to generate one or more samples), and generating one or more outputs, such as output data 212 from one or more inputs. Process 200 may be used to generate images or videos in various sensor types, not limited to, for example, RGB sensors, IR sensors, thermal sensors, etc. Process 100 may also include an attribute classifier 206 receiving one or more inputs, e.g., output data 212, and generating one or more outputs, e.g., attribute labels 220. Process 100 may also include a controllable generator 204 receiving one or more inputs, e.g., output data 212 and attribute labels 220, and generating one or more outputs, e.g., generated data 214, which may include at least a portion of the generated data set 132.

[0043] In at least one embodiment, at least one input leads to an unconditional generator 202 (unconditional generative model) in generating samples using an energy function that defines the semantics of the attributes that will be carried out by the samples. For example, an energy-based model (EBM) may be used to handle synthetic generation across a set of attributes. This may be at least in part due to the ability to combine energy functions in an EBM that represent different semantics to form a synthetic image generator. EBMs may be difficult to train in pixel space on high-resolution images, except that the formulation can be used in the latent space of a pre-trained generative model, e.g., an unconditional generator 202, making them extensible to high-resolution image generation. Non-restrictive examples of an unconditional generator 202 include one or more StyleGANs (Style Generative Adversarial Networks). In one or more embodiments, an unconditional generator 202 may include a mapping network to map points in the latent space to an intermediate latent space. The intermediate latent space can be used to control the style at each point in the generator model of the unconditional generator 202, and to control the introduction of noise as a source of variation at each point in the generator model.

[0044] Given a pre-trained generator, controllable generation can be obtained, at least partially, by training an attribute classifier 206, so that the controllable generator 204 can efficiently perform sampling in latent space. Whenever a new attribute is introduced, the energy function of the new attribute can be coupled with the existing energy function to form a new EBM without training a generative model from scratch. Such a plug-and-play method is relatively simple, quick to train, and the controllable generator 204 can be efficient at sampling, while having excellent performance in synthetic generation.

[0045] In one or more embodiments, the EBM may be constructed in a coupled space of data and attributes (e.g., a pre-trained GAN generator) where the marginal data distribution is indicated by an implicit distribution, and the conditional distribution of attributes, given data, is represented by an attribute classifier 206. Using reparameterization, the EBM formulation may derive a coupled energy function in this latent space (standard Gaussian distribution) where the latent distribution is known. The attribute classifier 206 may then only need to be trained in this data space by sampling performed by a controllable generator 204 in this latent space, for example, using an ordinary differential equation (ODE) solution method. Thus, adding controllability may only require training the attribute classifier 206 based on attribute semantics.

[0046] Referring here to Figure 3, each block of Method 300, and other methods described herein, include computational processes that can be performed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory. Methods may also be performed as computer-usable instructions stored in a computer storage medium. Methods may be provided, to name a few, by standalone applications, services, or hosted services (standalone or in combination with another hosted service), or by plugging into another product. Methods may be described in relation to specific systems and / or processes as examples. However, these methods may be performed additionally or alternatively by any one system or any combination of systems, including but not limited to those described herein.

[0047] Figure 3 is a flowchart illustrating a method 300 for controllably generating one or more samples based at least on evaluating one or more performance metrics of one or more attributes of one or more machine learning models, according to some embodiments of the present disclosure. Method 300 includes, in block B302, analyzing data corresponding to the output of at least one MLM. For example, an output analyzer 108 may analyze output data 124.

[0048] Method 300 includes evaluating one or more performance metrics in block B304. For example, the data set decisioner 112 may use performance data 128 to evaluate one or more performance metrics of one or more attributes of a sample applied to MLM 104 to generate output data 124, at least based on an analysis.

[0049] Method 300 includes identifying at least one value of at least one attribute in block B306. For example, the data set determination unit 112 may identify at least one value of at least one of one or more attributes based on evaluation.

[0050] Method 300 includes applying at least one input to one or more generating MLMs in block B308 to generate one or more samples corresponding to at least one value. For example, the data set determination unit 112 may apply control data 130 to the generating MLM 116, at least based on identification, to generate one or more samples corresponding to at least one value.

[0051] Method 300 includes training at least one MLM using one or more samples in block B310. For example, MLM trainer 120 may train MLM 104 at least on the basis of applying one or more samples to MLM 104.

[0052] Referring now to Figure 4, which is a flowchart illustrating a method 400 for controllably generating one or more samples of one or more attributes using one or more performance metric values, according to some embodiments of the present disclosure. Method 400 includes generating one or more performance metric values ​​associated with at least one MLM in block B402. For example, output analyzer 108 may generate one or more performance metric values ​​corresponding to one or more attributes of one or more objects depicted in a first one or more images applied to MLM 104.

[0053] Method 400 includes determining in block B404 that the performance of at least one MLM is below one or more thresholds for at least one attribute. For example, the data set determination 112 may use one or more performance metric values ​​to determine that the performance of MLM 104 is below one or more thresholds for at least one value of at least one of the attributes.

[0054] Method 400 includes applying at least one input to one or more generating MLMs in block B406 to generate one or more samples corresponding to at least one attribute. For example, the data set determination 112 may apply control data 130 to the generating MLM 116 to generate one or more samples corresponding to at least one value, based at least on the performance being below one or more thresholds for at least one value.

[0055] Method 400 includes training at least one MLM using one or more samples in block B408. For example, MLM trainer 120 may train MLM 104 using one or more samples.

[0056] Referring now to Figure 5, Figure 5 illustrates exemplary processes 500 that may be performed according to some embodiments of the present disclosure to controllably generate data for one or more machine learning models based on time-dependent patterns of one or more attributes.

[0057] Process 500 may be used to generate a time-series sequence of data items, such as video frames, which may form one or more samples of an expansion, training, validation, and / or test data set, for example, the generated data set 132 or other data sets described herein. In at least one embodiment, the data set determination 112 may assign frames to a sequence of frames of one or more attribute values ​​that will be depicted within the frames. The set of attribute values ​​may be determined based on at least one or more time-series scenarios that will be represented using one or more attribute values. For example, the data set determination 112 may map time-series scenarios to attribute values ​​in frames, which the generating MLM 116 may use to generate frames for depicting visual information corresponding to the attribute values.

[0058] Referring here to Figure 6, which illustrates one example of frames that may be generated to capture at least a portion of a time-series scenario according to some embodiments of the present disclosure. For example, Figure 6 shows a video 600 including a sequence of frames 610A, 610B, 610C, 610D, and 610D (also referred to as "frame 610") that may be generated using a generating MLM, e.g., generating MLM 116, with a set of attribute values ​​for assigning attribute values ​​to frames. In one or more embodiments, at least one of the frames of frame 610 may be generated using interpolation. Frame 610 may form an animation, where at least some of the attribute values ​​may be configured to animate one or more aspects of video 600 across frame 610. By example, but not limited to, the attribute values ​​of frame 610 include values ​​for head position and percentage of eyes closed. These attributes may be used to simulate one or more time-series scenarios across frame 310, e.g., drowsiness. For example, drowsiness may be simulated by controlling head position and blinking or other eye patterns to simulate dozing off. In at least one embodiment, MLM 104 and / or another MLM may be trained to predict or infer one or more aspects of a time-series scenario (e.g., existence, probability, level, etc.) using the simulated time-series scenario. As an example and not an limitation, process 500 may be used to identify time-series patterns and / or other characteristics of attributes related to a time-series scenario.

[0059] Figure 5 shows the components of the system, which may include a data set analyzer 502 and a frame attribute determiner 504. At a high level, process 500 may include the data set analyzer 502 receiving one or more inputs, e.g., one or more samples of a reference data set 508 (e.g., a real-world data set that may be known to represent one or more time-series scenarios), and generating one or more outputs, e.g., a time-series pattern 512 from one or more inputs. Process 500 may also include the frame attribute determiner 504 receiving one or more inputs, e.g., a time-series pattern 512, and generating one or more outputs, e.g., frame attributes 516 (e.g., of one or more frames) from one or more inputs. A generated data set 532 corresponding to the frame attributes 516 may be fabricated. In at least one embodiment, the frame attribute determiner 504 may be part of the data set determiner 112 in Figure 1. Therefore, the generated data set 532 may correspond to the generated data set 132 in Figure 1. However, in one or more embodiments, process 500 does not need to be used in conjunction with process 100. In any embodiment, the generating MLM 116 may be used to produce the generated data set 532.

[0060] In one or more embodiments, the reference data set 508 may include real-world data and / or simulated data. Not limited to, but as an example, the reference data set 508 may include fatigue or drowsiness data that can capture one or more time-series scenarios, e.g., sleepiness or drowsiness. For example, the reference data set 508 may include a set of frames forming a video (for example, over a length of one minute at 24 frames / second, 30 frames / second, 60 frames / second, etc.), where each set of frames is labeled with or otherwise associated with a time-series scenario and / or one or more values ​​of the time-series scenario. In this example, one or more values ​​may describe, be associated with, and / or be assigned to values, states, and / or levels corresponding to the Karolinska Sleepiness Scale (KSS), Epworth Sleepiness Scale (ESS), Stanford Sleepiness Scale (SSS), Johns Drowsiness Scale (JDS), and / or Observer Rated Drowsiness (ORD). The data set analyzer 502 may be used to determine one or more time-dependent patterns of one or more attributes corresponding to those time-dependent scenarios and / or values ​​so that it generates frame attributes 516 to simulate those aspects of the reference data set 508 using a generating MLM. For example, the frame attribute determination unit 504 may determine the frame attributes 516 of a frame using a time-series pattern 512 associated by the data set analyzer 502 with a KSS level of 8 and / or an ORD state of "not sleepy" in order to generate video 600. Video 600 may then be used to train one or more MLMs, such as MLM 104, to predict KSS levels and / or ORD states using the video.In one or more embodiments, the KSS level and / or ORD state used to determine the frame attribute 516 may be used as the ground truth of the video.

[0061] Figure 5 shows non-limiting examples of time-series patterns 512, which include blink rate 512A, eye-closed percentage 512B, blink amplitude 512C, yawn frequency 512D, blink duration 512E, and eye velocity 212F. Referring now to Figure 7, Figure 7 is an illustration used to illustrate examples of how one or more of the time-series patterns 512 may be extracted from the reference data set 508 according to some embodiments of the present disclosure.

[0062] Figure 7 shows frames 700A and 700B of frame 700, which may belong to the video of reference data set 508 and can be analyzed using the data set analyzer 502 to determine one or more of the time-series patterns 512. In at least one embodiment, the data set analyzer 502 may determine one or more landmarks within frame 700 and measure or otherwise track or monitor one or more landmarks across one or more of the frames 700 to determine one or more time-series patterns. Examples of landmarks include landmark points P1, P2, P3, P4, P6, and P6, which can be measured and / or evaluated to determine one or more attributes (e.g., time-series patterns across the frames) of one or more of the frames. Landmark points can be determined using any suitable technique, e.g., a landmark MLM trained to identify landmarks.

[0063] In at least one embodiment, the data set analyzer 502 can determine one or more attribute values ​​of one or more frames, and then analyze the attribute values ​​over time to determine one or more patterns over time. For example, Figure 7 shows a graph 704 of attribute values ​​710 over time. In at least one embodiment, the attribute values ​​710 can be calculated by the data set analyzer 502 using equation (1):

number

[0064] The data set analyzer 502 may use the attribute value 710 to calculate the blink rate 512A as the number of blinks per unit second, and frames 3 and 7 are examples of start and end times that the data set analyzer 502 can identify for blinks from the attribute value 710. As another example, the data set analyzer 502 may use the attribute value 710 to calculate the closed-eye percentage 512B as the percentage of frames 700, e.g., frame 700B, in which the attribute value 710 indicates a closed eye. The data set analyzer 502 may use the attribute value 710 to calculate the blink amplitude 512C based at least on the measured duration 720 of the blink. For example, the blink amplitude 512C can be calculated using equation (2):

number

[0065] The data set analyzer 502 may use attribute values ​​710 to calculate the eye velocity 512F based on at least the measured start, lower, and end frames of the blink. For example, the eye velocity 512F may be calculated using equation (3):

number

[0066] Similar techniques may be used to determine other time-series patterns, for example, by using head poses for nodding patterns or mouth landmarks for yawning patterns. Although the disclosed embodiments are primarily described with respect to samples corresponding to frames or images, the disclosed techniques may be applied to other forms of samples and / or time-series sequences, such as audio samples, multimedia samples, and / or any form of samples that can be applied as input to MLMs.

[0067] Referring here to Figure 8, which is a flowchart illustrating a method 800 for controllably generating one or more samples based at least on associating one or more time-series patterns with one or more time-series scenarios, according to some embodiments of the present disclosure. Method 800 includes associating one or more time-series patterns of one or more attributes with one or more time-series scenarios in block B802. For example, a data set analyzer 502 may analyze a reference data set 508 of time-series scenarios (and / or their values) to determine time-series patterns 512 associated with time-series scenarios.

[0068] Method 800 includes assigning values ​​of one or more attributes to a sample sequence on at least one basis that one or more time-series attributes are associated with one or more time-series scenarios in block B804. For example, a frame attribute determiner 504 may determine the frame attributes 516 of frames in one or more sequences using a time-series pattern 512 associated with a time-series scenario and assign the frame attributes 516 to each frame of the frames in one or more sequences.

[0069] Method 800 includes generating a sequence of samples using assigned values ​​in block B806 using one or more generating MLMs. For example, a generating MLM, such as generating MLM 116, may be used to generate one or more sequences of frames.

[0070] Method 800 includes training at least one MLM using a sequence of samples in block B808. For example, MLM104 and / or another MLM may be trained using one or more sequences of frames.

[0071] Figure 9 illustrates an exemplary process 900 that may be used for attribute-controllable generation of one or more data sets according to some embodiments of the present disclosure. Process 900 provides a general technique that may be used for attribute-controllable generation of one or more data sets. In various examples, process 900 may be used to generate one or more samples of a generated data set 932 for expansion, testing, training (initial training and / or retraining), and / or validation. For example, process 900 includes input data 902 being provided to a data set determination unit 112 to generate control data 130, and the control data 130 being applied to a generation MLM 116 to produce the generated data set 932.

[0072] As shown in Figure 1, process 900 may be included in process 100, where input data 902 includes performance data 128 and generated data set 932 includes generated data set 132. Also, as shown in Figure 5, process 900 may be included in process 500, where input data 902 includes reference data set 508 and / or time-series pattern 512 and generated data set 532 includes generated data set 132. For example, data set determiner 112 may include data set analyzer 502 and / or frame attribute determiner 504. Process 500 includes time-series pattern, but process 500 may or may not include time-series pattern 512, and may more generally include data set analyzer 502 and reference data set 508 to determine one or more characteristics of the reference data set (e.g., attribute distribution, attribute present, corner case, etc.). Furthermore, the frame attribute determination unit 504 may more commonly be used to determine one or more attributes of one or more samples based on at least one or more characteristics.

[0073] As described herein, input data 902 may include and / or indicate a set of attributes that will be included in the generated data set 932. Performance data 128 is an example of input data 902 that indicates a set of attributes that will be included in the generated data set 932. As a further example, input data 902 may include a list of attributes. For example, in a compliance test, a set of attributes that will be used for the test is often provided. In other examples, input data 902 may include test scenarios that exist in only a few (e.g., within one data set), such as driver drowsiness levels that reach KSS9. Using the disclosed techniques, data set classifier 112 may classify control data 130 to produce a generated data set 932 that includes a set of attributes. For example, data set classifier 112 may use generating MLM 116 to generate a data set according to a desirable distribution of attributes and / or combination of attributes for training, validation, and / or test data sets and / or one or more of those data sets for the entire data set from which it is formed. One example of such a distribution includes a uniform distribution of attributes and / or combinations of attributes. Using such techniques, a dataset can be customized to suit specific scenarios related to the attributes.

[0074] In addition or by alternative means, the data set classifier 112 may use the generating MLM 116 to construct the generated data set 932, and to define and / or modify the distribution of attributes and / or attribute combinations in the data set to be used for augmentation, training, validation, and / or testing. As an example, and not an limitation, at least 80% of the data may be reserved for training (e.g., 95%), and any remaining data may be used for validation and / or testing (e.g., 5%). Validation and / or testing data sets may be augmented using a GNN to include one or more additional samples corresponding to attributes and / or combinations thereof that exist in small numbers to contain sufficient testing data for those scenarios.

[0075] In at least one embodiment, the input data 902 may include one or more samples, which may include one or more subjects (e.g., different faces, which may be randomly generated and / or from real-world data) of one or more classes. The data set classifier 112 may modify one or more attributes of each subject to produce corresponding samples for the generated data set. For example, for a set of subjects and attributes, the data set classifier 112 may use the generating MLM 116 to produce samples of each possible combination of attributes for each subject. The generated data set 932 may be used for expansion, testing, training (initial training and / or retraining), and / or validation. For example, the MLM 104 may be initially trained using subjects and then trained using the generated data set 932, or the MLM 104 may be first trained using at least one or more samples from the generated data set 932.

[0076] Referring now to Figure 10, which is a flowchart illustrating a method 1000 for controllably generating one or more samples based on at least analysis of input data, according to some embodiments of the present disclosure. Method 1000 includes receiving input data in block B1002. For example, a data set determination unit 112 may receive input data 902. Method 1000 includes analyzing input data in block B1004. For example, the data set determination unit 112 may analyze the input data 902. Method 1000 includes generating one or more samples based on the analysis using one or more generating MLMs in block B1006. For example, the data set determination unit 112 may generate one or more samples of the generated data set 932 using the generating MLM 116, based on the analysis. Method 1000 includes applying one or more of the samples to at least one MLM in block B1008. For example, one or more of the samples may be applied to MLM104 for training, validation, and / or testing.

[0077] (Example autonomous vehicle) Figure 11A illustrates exemplary autonomous vehicle 1100 according to several embodiments of the present disclosure. Autonomous vehicle 1100 (or, as referred to herein as "vehicle 1100") may include, but is not limited to, passenger vehicles such as passenger cars, trucks, buses, first responder vehicles, shuttle buses, electric or motorized bicycles, motorcycles, fire engines, police vehicles, ambulances, boats, construction vehicles, submarines, drones, trailer-mounted vehicles, and / or other types of vehicles (e.g., unmanned and / or carrying one or more passengers). Autonomous vehicles are generally described in terms of automation levels as defined by the National Highway Traffic Safety Administration (NHTSA), departments of the U.S. Department of Transportation, and the Society of Automotive Engineers (SAE) "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicle" (standard number J3016-201806, published June 15, 2018; standard number J3016-201609, published September 30, 2016; and previous and future versions of this standard). Vehicle 1100 may have the capability to perform functions at one or more levels of autonomous driving from Level 3 to Level 5. Vehicle 1100 may have the capability to perform functions at one or more levels of autonomous driving from Level 1 to Level 5. For example, depending on the embodiment, the vehicle 1100 may have the capability of driver assistance (Level 1), partial automation (Level 2), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5). In this specification, the term “autonomous” may include any and / or all types of autonomy of the vehicle 1100 or any other machine, such as being fully autonomous, highly autonomous, conditionally autonomous, partially autonomous, providing auxiliary autonomy, semi-autonomous, primarily autonomous, or other designations.

[0078] Vehicle 1100 may include components such as the vehicle's chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components. Vehicle 1100 may include a propulsion system 1150, such as an internal combustion engine, a hybrid power unit, a fully electric engine, and / or another propulsion system type. The propulsion system 1150 may be connected to the vehicle 1100's drivetrain, which may include a transmission, to enable propulsion for the vehicle 1100. The propulsion system 1150 may be controlled in response to receiving signals from a throttle / accelerator 1152.

[0079] A steering system 1154, which may include a steering wheel, may be used to steer the vehicle 1100 (for example, along a desired course or route) when the propulsion system 1150 is operating (for example, when the vehicle is moving). The steering system 1154 may receive signals from the steering actuator 1156. The steering wheel may also be an option for fully automated (level 5) functionality.

[0080] The brake sensor system 1146 may be used to operate the vehicle brakes in response to receiving signals from the brake actuator 1148 and / or the brake sensor.

[0081] The controller 1136, which may include one or more system-on-a-chip (SoC) 1104 (Figure 11C) and / or GPUs, can provide signals (e.g., expressions of commands) to one or more components and / or systems of the vehicle 1100. For example, the controller can send signals to actuate the vehicle brakes via one or more brake actuators 1148, actuate the steering system 1154 via one or more steering actuators 1156, and actuate the propulsion system 1150 via one or more throttle / accelerators 1152. The controller 1136 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operational commands (e.g., signals representing commands) to enable autonomous driving and / or assist the driver in driving the vehicle 1100. The controller 1136 may include a first controller 1136 for autonomous driving functions, a second controller 1136 for functional safety functions, a third controller 1136 for artificial intelligence functions (e.g., computer vision), a fourth controller 1136 for infotainment functions, a fifth controller 1136 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 1136 may handle two or more of the aforementioned functions, and two or more controllers 1136 may handle a single function, and / or any combination thereof.

[0082] The controller 1136 can provide signals for controlling one or more components and / or systems of the vehicle 1100 in response to sensor data (e.g., sensor inputs) received from one or more sensors. Sensor data may be received from, but not limited to, examples of, global navigation satellite system sensors 1158 (e.g., global positioning system sensors), RADAR sensors 1160, ultrasonic sensors 1162, LIDAR sensors 1164, inertial measurement unit (IMU) sensors 1166 (e.g., accelerometers, gyroscopes, magnetic compasses, magnetometers, etc.), microphones 1196, stereo cameras 1168, wide-view cameras 1170 (e.g., fisheye cameras), infrared cameras 1172, surround cameras 1174 (e.g., 360-degree cameras), long-range and / or medium-range cameras 1198, speed sensors 1144 (e.g., for measuring the speed of a vehicle 1100), vibration sensors 1142, steering sensors 1140, brake sensors (e.g., as part of a brake sensor system 1146), and / or other sensor types.

[0083] One or more of the controllers 1136 may receive inputs (represented, for example, by input data) from the instrument cluster 1132 of the vehicle 1100 and provide outputs (represented, for example, by output data, display data, etc.) via a human-machine interface (HMI) display 1134, an audible annunciator, a loudspeaker, and / or other components of the vehicle 1100. The outputs may include information such as vehicle speed, speed, time, map data (e.g., HD map 1122 in Figure 11C), location data (e.g., the location of the vehicle 1100, such as on a map), direction, the location of other vehicles (e.g., occupied grid), and information about objects and the status of objects as perceived by the controller 1136. For example, the HMI display 1134 may display information regarding the presence of one or more objects (e.g., road signs, warning signs, changes in traffic signals, etc.) and / or driving operations that the vehicle has performed, is performing, or will perform (e.g., changing lanes now, exiting exit 34B within 3.22 km (2 miles), etc.).

[0084] Vehicle 1100 further includes a network interface 1124 that can communicate over one or more networks using one or more wireless antennas 1126 and / or a modem. For example, the network interface 1124 may have the capability to communicate over LTE, WCDMA®, UMTS, GSM, CDMA2000, etc. The wireless antennas 1126 can also enable communication between objects in the environment (e.g., vehicles, mobile devices, etc.) using local area networks such as Bluetooth®, Bluetooth® LE, Z-Wave, ZigBee, and / or low-power wide-area networks (LPWANs) such as LoRaWAN, SigFox.

[0085] Figure 11B shows examples of camera positions and fields of view of the exemplary autonomous vehicle 1100 of Figure 11A according to several embodiments of the present disclosure. The cameras and their respective fields of view are exemplary embodiments and are not intended to limit the scope. For example, additional and / or alternative cameras may be included, and / or cameras may be placed in different locations on the vehicle 1100.

[0086] The camera type may include, but is not limited to, a digital camera that can be used with components and / or systems of the vehicle 1100. The camera may operate at Automotive Safety Integrity Level (ASIL) B and / or other ASILs. Depending on the embodiment, the camera type may have the capability of any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc. The camera may have the capability to use a roll shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include an RCCC (red clear clear clear) color filter array, an RCCB (red clear clear blue) color filter array, an RBGC (red blue green clear) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, clear pixel cameras, such as cameras having RCCC, RCCB, and / or RBGC color filter arrays, may be used in efforts to increase light sensitivity.

[0087] In some applications, one or more cameras may be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function mono-camera may be installed to provide functions including lane departure warning, traffic sign assist, and intelligent headlamp control. One or more cameras (e.g., all cameras) may simultaneously record and provide image data (e.g., video).

[0088] One or more of the cameras may be mounted in custom-designed (3D-printed) mounting parts to eliminate stray light and reflections from inside the vehicle (e.g., reflections from the dashboard reflected in the windshield mirror) that may interfere with the camera's image data capture capability. Referring to side mirror mounting parts, the side mirror parts may be custom 3D-printed so that the camera mounting plate conforms to the shape of the side mirror. In some examples, the camera may be integrated within the side mirror. For side-view cameras, the camera may also be integrated within four struts located at each corner of the cabin.

[0089] A camera having a field of view that includes a portion of the environment in front of the vehicle 1100 (e.g., a forward-facing camera) may be used for surround view to help identify forward paths and obstacles and, with the help of one or more controllers 1136 and / or control SoCs, provide information essential for generating an occupied grid and / or determining a preferred vehicle path. The forward-facing camera may also be used to perform many of the same ADAS functions as LIDAR, including emergency braking, pedestrian detection, and collision avoidance. The forward-facing camera may also be used for ADAS functions and systems, including other functions such as lane departure warning (LDW), autonomous cruise control (ACC), and / or traffic sign recognition.

[0090] Various cameras may be used in forward-facing configurations, including, for example, a monocular camera platform including a CMOS (complementary metal oxide semiconductor) color imaging device. Another example may be a wide-view camera 1170, which can be used to capture objects that come into view from the periphery (e.g., pedestrians, crossing traffic, or bicycles). Although only one wide-view camera is shown in Figure 11B, any number of wide-view cameras 1170 may be present in the vehicle 1100. In addition, long-range cameras 1198 (e.g., a long-view stereo camera pair) may be used for depth-based object detection, particularly for objects for which the neural network has not yet been trained. Long-range cameras 1198 may also be used for object detection and classification, as well as basic object tracking.

[0091] One or more stereo cameras 1168 may also be included in a forward-facing configuration. The stereo camera 1168 may include an integrated control unit with an expandable processing unit that may provide a programmable logic (FPGA) and a multi-core microprocessor with an integrated CAN or Ethernet® interface on a single chip. Such a unit may be used to generate a 3D map of the vehicle's environment, including distance estimates of all points in the image. An alternative stereo camera 1168 may include a compact stereo vision sensor that includes two camera lenses (one on the left and one on the right) and an image processing chip that can measure the distance from the vehicle to an object and activate autonomous emergency braking and lane departure warning functions using the generated information (e.g., metadata). Other types of stereo cameras 1168 may be used in addition to or instead of those described herein.

[0092] A camera having a field of view including a portion of the environment on the sides of the vehicle 1100 (e.g., a side-view camera) may be used for surround view, providing information used to create and update the occupy grid and generate a side impact collision warning. For example, surround cameras 1174 (e.g., four surround cameras 1174 as shown in Figure 11B) may be positioned on the vehicle 1100. The surround cameras 1174 may include wide-view cameras 1170, fisheye cameras, 360-degree cameras, and / or similar. For example, four fisheye cameras may be positioned in front of, behind, and on the sides of the vehicle. In an alternative configuration, the vehicle may use three surround cameras 1174 (e.g., left, right, and rear) and utilize one or more other cameras (e.g., forward-facing cameras) as a fourth surround view camera.

[0093] A camera having a field of view that includes a portion of the environment behind the vehicle 1100 (e.g., a rear-view camera) may be used for parking assistance, surround view, rear collision warning, and creation and updating of the occupancy grid. A wide variety of cameras may be used, including, but not limited to, cameras also suitable as forward-facing cameras (e.g., long-range and / or medium-range camera 1198, stereo camera 1168, infrared camera 1172, etc.) as described herein.

[0094] Figure 11C is a block diagram of an exemplary system architecture of the exemplary autonomous vehicle 1100 of Figure 11A, according to some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted together. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in combination with other components, and in any appropriate combination and location. The various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory.

[0095] Each component, feature, and system of the vehicle 1100 in Figure 11C is illustrated as being connected via a bus 1102. The bus 1102 may include a Controller Area Network (CAN) data interface (or, as herein referred to, the “CAN bus”). The CAN may also be a network within the vehicle 1100 used to assist in the control of various features and functions of the vehicle 1100, such as the operation of brakes, acceleration, braking, steering, and windshield wipers. The CAN bus may be configured to have dozens or hundreds of nodes, each having its own unique identifier (e.g., CAN ID). The CAN bus may be read to find steering angle, ground speed, engine revolutions per minute (RPM), button position, and / or other vehicle condition indicators. The CAN bus may be ASIL B compliant.

[0096] Bus 1102 is described herein as a CAN bus, but this is not intended to limit it. For example, FlexRay and / or Ethernet® may be used in addition to or as an alternative to a CAN bus. In addition, a single line is used to represent bus 1102, but this is not intended to limit it. There may be any number of buses 1102, which may include, for example, one or more CAN buses, one or more FlexRay buses, one or more Ethernet® buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 1102 may be used to perform different functions and / or for redundancy. For example, a first bus 1102 may be used for collision avoidance functions, and a second bus 1102 may be used for operation control. In any example, each bus 1102 may communicate with any of the components of vehicle 1100, and two or more buses 1102 may communicate with the same component. In some examples, each SoC 1104, each controller 1136, and / or each computer within the vehicle may have access to the same input data (e.g., input from sensors in the vehicle 1100) and may be connected to a common bus such as a CAN bus.

[0097] The vehicle 1100 may include one or more controllers 1136, such as those described herein with respect to Figure 11A. The controllers 1136 may be used for a variety of functions. The controllers 1136 may be connected to any of the various other components and systems of the vehicle 1100 and may be used for the control of the vehicle 1100, the artificial intelligence of the vehicle 1100, infotainment for the vehicle 1100, and / or the like.

[0098] Vehicle 1100 may include a system-on-a-chip (SoC) 1104. The SoC 1104 may include a CPU 1106, a GPU 1108, a processor 1110, a cache 1112, an accelerator 1114, a data store 1116, and / or other components and features not shown. The SoC 1104 may be used to control vehicle 1100 in various platforms and systems. For example, the SoC 1104 may be coupled in a system (e.g., a system of vehicle 1100) that has an HD map 1122 that can obtain map refreshes and / or updates from one or more servers (e.g., server 1178 in Figure 11D) via a network interface 1124.

[0099] The CPU 1106 may include a CPU cluster or CPU complex (or, as referred to herein, "CCPLEX"). The CPU 1106 may include multiple cores and / or L2 caches. For example, in some embodiments, the CPU 1106 may include eight cores in a coherent multiprocessor configuration. In some embodiments, the CPU 1106 may include four dual-core clusters, each cluster having its own dedicated L2 cache (e.g., 2MBL2 cache). The CPU 1106 (e.g., CCPLEX) may be configured to support concurrent cluster operation, which allows any combination of the clusters of the CPU 1106 to be active at a given time.

[0100] The CPU1106 can implement power management capabilities that include one or more of the following features: individual hardware blocks may be automatically clock-gated when idle to conserve dynamic power; each core clock may be gated when a core is not actively executing instructions by executing WFI / WFE instructions; each core may be independently power-gated; each core cluster may be independently clock-gated when all cores are clock-gated or power-gated; and / or each core cluster may be independently power-gated when all cores are power-gated. The CPU1106 can further implement enhanced algorithms for managing power states, where acceptable power states and expected wake-up times are specified, and the hardware / microcode determines the best power state to input to the cores, clusters, and CCPLEX. The processing core may support a simplified power state input sequence in software where the work is offloaded to the microcode.

[0101] The GPU1108 may include an integrated GPU (or, as referred to herein, "iGPU"). The GPU1108 may be programmable and efficient for parallel workloads. In some embodiments, the GPU1108 may be able to use an enhanced tensor instruction set. The GPU1108 may include one or more streaming microprocessors, each of which may include an L1 cache (e.g., an L1 cache with a storage capacity of at least 96KB), and two or more of the streaming microprocessors may share an L2 cache (e.g., an L2 cache with a storage capacity of 512KB). In some embodiments, the GPU1108 may include at least eight streaming microprocessors. The GPU1108 may be able to use a Computation Application Programming Interface (API). In addition, the GPU1108 may be able to use one or more parallel computing platforms and / or programming models (e.g., NVIDIA® CUDA).

[0102] The GPU1108 can be power-optimized for best performance in automotive and embedded use cases. For example, the GPU1108 can be manufactured on a FinFET (Fin field-effect transistor). However, this is not intended to be limiting, and the GPU1108 can be manufactured using other semiconductor manufacturing processes. Each streaming microprocessor can incorporate several mixed-precision processing cores divided into multiple blocks. As an example, and not an limitation, 64 PF32 cores and 32 PF64 cores may be divided into four processing blocks. In such an example, each processing block may be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, 2 mixed-precision NVIDIA TENSOR COREs for deep learning matrix operations, an L0 instruction cache, a warp scheduler, a dispatch unit, and / or a 64KB register file. In addition, streaming microprocessors may include independent, concurrent integer and floating-point data paths to provide efficient execution of workloads involving a mixture of computation and addressing operations. Streaming microprocessors may include independent thread scheduling capabilities to enable finer-grained synchronization and coordination between concurrent threads. Streaming microprocessors may include coupled L1 data caches and shared memory units to improve performance while simplifying programming.

[0103] In some examples, the GPU1108 may include high-bandwidth memory (HBM) and / or a 16GB HBM2 memory subsystem to provide a peak memory bandwidth of 900 GB / s. In some examples, in addition to or instead of HBM memory, synchronous graphics random-access memory (SGRAM), such as graphics double data rate type five synchronous random-access memory (GDDR5), may be used.

[0104] The GPU1108 can incorporate unified memory technology, including access counters, to enable more precise movement of memory pages to the processor that most frequently accesses them, thereby improving the efficiency of shared memory ranges across processors. In some examples, address translation service (ATS) support can be used to allow the GPU1108 to directly access the CPU1106 page table. In such examples, when the GPU1108 memory management unit (MMU) experiences a miss, an address translation request can be sent to the CPU1106. In response, the CPU1106 can look up its page table for virtual-to-real-address mapping and send the translation back to the GPU1108. As such, unified memory technology can enable a single, unified virtual address space for both the CPU1106 and GPU1108 memory, thereby simplifying GPU1108 programming and porting of applications to the GPU1108.

[0105] In addition, the GPU1108 may include an access counter that can record how often the GPU1108 accesses the memory of other processors. The access counter can help ensure that memory pages are moved to the physical memory of the processor that accesses that page most frequently.

[0106] The SoC1104 may include any number of caches 1112, including those described herein. For example, cache 1112 may include an L3 cache available to both the CPU 1106 and the GPU 1108 (e.g., connected to both the CPU 1106 and the GPU 1108). Cache 1112 may include a write-back cache capable of recording line states, for example, by using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). The L3 cache may include 4 MB or more, depending on the embodiment, although a smaller cache size may be used.

[0107] The SoC1104 may include an arithmetic logic unit (ALU) that can be used to perform processing for any of the various tasks or operations of the vehicle 1100 (for example, a processing DNN). In addition, the SoC1104 may include a floating-point unit (FPU) (or other mass coprocessor or numerical coprocessor type) for performing mathematical operations within the system. For example, the SoC104 may include one or more FPUs integrated as execution units within the CPU1106 and / or GPU1108.

[0108] The SoC1104 may include one or more accelerators 1114 (e.g., a hardware accelerator, a software accelerator, or a combination thereof). For example, the SoC1104 may include a hardware acceleration cluster that may include an optimized hardware accelerator and / or a large on-chip memory. The large on-chip memory (e.g., 4MB of SRAM) may enable the hardware acceleration cluster to accelerate neural networks and other computations. The hardware acceleration cluster may be used to complement the GPU1108 and to offload some of the tasks of the GPU1108 (e.g., to free up more cycles of the GPU1108 to perform other tasks). As an example, the accelerator 1114 may be used for target workloads that are sufficiently stable to be suitable for acceleration (e.g., perception, convolutional neural networks (CNNs), etc.). In this specification, the term "CNN" may include all types of CNNs, including region-based or regional convolutional neural networks (RCNNs) and fast RCNNs (for example, as used for object detection).

[0109] The accelerator 1114 (e.g., a hardware acceleration cluster) may include a deep learning accelerator (DLA). The DLA may include one or more tensor processing units (TPUs) that can be configured to provide an additional 10 trillion operations per second for deep learning applications and inference. The TPU may also be an accelerator configured and optimized to perform image processing functions (e.g., CNN, RCNN, etc.). The DLA may further be optimized for a specific set of neural network types and floating-point operations, as well as for inference. The design of the DLA can provide more performance per millisecond than a general-purpose GPU and far exceed the performance of a CPU. The TPU can perform several functions, including, for example, single-instance convolution functions that support INT8, INT16, and FP16 data types for both features and weights, as well as post-processing functions.

[0110] DLA can quickly and efficiently run neural networks, particularly CNNs, on processed or unprocessed data for any of a variety of functions, including but not limited to: CNNs for object recognition and detection using data from camera sensors; CNNs for distance estimation using data from camera sensors; CNNs for emergency vehicle detection, identification, and detection using data from microphones; CNNs for facial recognition and vehicle owner identification using data from camera sensors; and / or CNNs for security and / or safety-related events.

[0111] DLA can perform any function of GPU1108, and by using inference accelerators, for example, a designer can target either DLA or GPU1108 for any function. For example, a designer can focus on CNN and floating-point arithmetic processing on DLA, and leave other functions to GPU1108 and / or other accelerators 1114.

[0112] The accelerator 1114 (for example, a hardware accelerator cluster) may include a programmable vision accelerator (PVA), which may be referred to herein as a computer vision accelerator. A PVA may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. A PVA can provide a balance between performance and flexibility. For example, each PVA may, but is not limited to, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors.

[0113] A RISC core can interact with an image sensor (for example, the image sensor of one of the cameras described herein), an image signal processor, and / or similar devices. Each RISC core may include any amount of memory. Depending on the embodiment, a RISC core may use one of several protocols. In some examples, a RISC core can run a real-time operating system (RTOS). A RISC core may be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, a RISC core may include an instruction cache and / or tightly coupled RAM.

[0114] DMA can enable PVA components to access system memory independent of the CPU 1106. DMA can support any number of features used to bring optimizations to the PVA, including but not limited to supporting multidimensional addressing and / or circular addressing. In some examples, DMA can support up to six or more dimensions of addressing, which may include block width, block height, block depth, horizontal block stepping, vertical block stepping, and / or depth stepping.

[0115] A vector processor may also be a programmable processor that can be designed to efficiently and flexibly execute the programming of computer vision algorithms and provide signal processing capabilities. In some examples, a PVA may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, a DMA engine (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem can act as the primary processing engine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or vector memory (e.g., VMEM). The VPU core may include a digital signal processor, such as a single-instruction, multiple-data (SIMD), or very-long instruction word (VLIW) digital signal processor. A combination of SIMD and VLIW can increase throughput and speed.

[0116] Each vector processor may include an instruction cache and be linked to dedicated memory. As a result, in some examples, each vector processor may be configured to run independently of other vector processors. In other examples, the vector processors included in a particular PVA may be configured to use data parallelism. For example, in some embodiments, multiple vector processors included in a single PVA may execute the same computer vision algorithm, but on different regions of an image. In other examples, the vector processors included in a particular PVA may execute different computer vision algorithms simultaneously on the same image, or even execute different algorithms sequentially on the image or parts of an image. In particular, any number of PVAs may be included in a hardware acceleration cluster, and any number of vector processors may be included in each PVA. In addition, a PVA may include additional error correction code (ECC) memory to enhance overall system safety.

[0117] The accelerator 1114 (e.g., a hardware accelerator cluster) may include a computer vision network on-chip and SRAM to provide high-bandwidth, low-latency SRAM for the accelerator 1114. In some examples, the on-chip memory may include at least 4 MB of SRAM consisting of eight field-configurable memory blocks, which may be accessible by both the PVA and DLA, but not limited to examples. Each pair of memory blocks may include an advanced peripheral bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory may be used. The PVA and DLA can access the memory via a backbone that provides the PVA and DLA with high-speed access to the memory. The backbone may include a computer vision network on-chip that interconnects the PVA and DLA to the memory (e.g., using an APB).

[0118] A computer vision network on-chip may include an interface that determines whether both the PVA and DLA are activatable and enable signals before any control signals / addresses / data are transmitted. Such an interface can provide separate phases and separate channels for transmitting control signals / addresses / data, as well as burst-type communication for continuous data transfer. This type of interface may conform to ISO 26262 or IEC 61508 standards, but other standards and protocols may be used.

[0119] In some embodiments, the SoC1104 may include a real-time ray tracing hardware accelerator, such as the one described in Patent Document 1, filed August 10, 2018. The real-time ray tracing hardware accelerator may be used to quickly and efficiently determine the location and size of objects (e.g., in a world model) to generate real-time visualization simulations for RADAR signal interpretation, acoustic propagation synthesis and / or analysis, SONAR system simulation, general wave propagation simulation, comparison to LIDAR data for localization and / or other functions, and / or other uses. In some embodiments, one or more tree traversal units (TTUs) may be used to perform one or more ray tracing-related operations.

[0120] The accelerator 1114 (e.g., a hardware accelerator cluster) has diverse applications for autonomous driving. The PVA may also be a programmable vision accelerator that can be used in critical processing stages in ADAS and autonomous vehicles. The capabilities of the PVA are suitable for areas of algorithms that require predictable processing at low power and low latency. In other words, the PVA works well in semi-high density or high density normal computations, even on small data sets, where predictable execution time is required along with low latency and low power. Therefore, since the PVA is efficient in operation in object detection and integer computation, in relation to a platform for autonomous vehicles, the PVA is designed to run classic computer vision algorithms.

[0121] For example, according to one embodiment of this technology, PVA is used to perform computer stereo vision. Semi-global matching-based algorithms may be used in some examples, but this is not intended to be limiting. Numerous applications for Level 3-5 autonomous driving require motion estimation / stereo matching on the fly (e.g., SFM (structure from motion), pedestrian recognition, lane detection, etc.). PVA can perform computer stereo vision functions with input from two monocular cameras.

[0122] In some applications, PVA can be used to perform high-density optical flow by processing raw RADAR data (e.g., using 4D Fast Fourier Transform) to provide processed RADAR data. In other applications, PVA is used for time-of-flight depth processing, for example, by processing raw time-of-flight data to provide processed time-of-flight data.

[0123] DLA can be used to run any type of network to enhance control and driving safety, for example, a neural network that outputs a confidence value for each object detection. Such confidence values ​​can be interpreted as probabilities or as providing the relative "weight" of each detection compared to further detections. This confidence value allows the system to make other decisions about which detections should be considered true positives rather than false positives. For example, the system can set a confidence threshold and consider only detections that exceed the threshold as true positives. In an automatic emergency braking (AEB) system, a false positive detection would cause the vehicle to automatically perform emergency braking, which is obviously undesirable. Therefore, only the most confident detections should be considered as triggers for AEB. DLA can run a neural network that devolves the confidence values. The neural network can accept at least a subset of parameters as its input, such as bounding box dimensions, ground plane estimation acquired (e.g., from another subsystem), vehicle orientation, distance, inertial measurement unit (IMU) sensor output correlated with 3D position estimation of the object acquired from the neural network and / or other sensors (e.g., LIDAR sensor 1164 or RADAR sensor 1160), and others.

[0124] The SoC1104 may include a data store 1116 (for example, memory). The data store 1116 may also be the on-chip memory of the SoC1104 and can store neural networks that will run on the GPU and / or DLA. In some examples, the data store 1116 may have a capacity large enough to store multiple instances of the neural network for redundancy and safety. The data store 1112 may comprise an L2 or L3 cache 1112. References to the data store 1116 may include references to memory associated with the PVA, DLA, and / or other accelerators 1114, as described herein.

[0125] The SoC1104 may include one or more processors 1110 (e.g., embedded processors). The processors 1110 may include a boot and power management processor, which may be a dedicated processor and subsystem for handling boot power and management capabilities and associated security enforcement. The boot and power management processor may also be part of the SoC1104 boot sequence and can provide runtime power management services. The boot power and management processor may provide clock and voltage programming, assistance with system low-power state transitions, management of SoC1104 thermal and temperature sensors, and / or management of SoC1104 power states. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to temperature, and the SoC1104 may use the ring oscillators to detect the temperatures of the CPU 1106, GPU 1108, and / or accelerator 1114. If the temperature is determined to have exceeded a threshold, the boot and power management processor may enter a temperature fault routine, placing the SoC1104 into a lower power state and / or putting the vehicle 1100 into a driver-assistance mode for safe shutdown (for example, safely shutting down the vehicle 1100).

[0126] Processor 1110 may further include a set of integrated processors capable of performing the functions of an audio processing engine. The audio processing engine may also be an audio subsystem enabling full hardware support for multi-channel audio through multiple interfaces and a wide and flexible range of audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core having a digital signal processor with dedicated RAM.

[0127] The processor 1110 may further include an always-on processor engine that can provide the necessary hardware features to support low-power sensor management and wake use cases. The always-on processor engine may include a processor core, tightly coupled RAM, support peripherals (e.g., timer and interrupt controllers), various I / O controller peripherals, and routing logic.

[0128] The processor 1110 may further include a safety cluster engine, which includes a dedicated processor subsystem for handling safety management in automotive applications. The safety cluster engine may include two or more processor cores, tightly coupled RAM, supporting peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In safety mode, the two or more cores may operate in lockstep mode and function as a single core with comparison logic for detecting any differences between their operations.

[0129] The processor 1110 may further include a real-time camera engine, which may include a dedicated processor subsystem for handling real-time camera management.

[0130] The processor 1110 may further include a high dynamic range signal processor, which may include an image signal processor, a hardware engine that is part of the camera processing pipeline.

[0131] The processor 1110 may include a video image synthesizer, which may also be a processing block (for example, implemented on a microprocessor) that implements post-video processing functions required by the video playback application to produce the final image for the player window. The video image synthesizer can perform lens distortion correction on the wide-view camera 1170, the surround camera 1174, and / or the in-cabin surveillance camera sensors. The in-cabin surveillance camera sensors are preferably monitored by a neural network running on another instance of the advanced SoC, configured to identify and appropriately respond to in-cabin events. The in-cabin system can perform lip-reading to activate cellular services and make phone calls, transcribe emails, change the vehicle's destination, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. Certain functions are available to the driver only when operating in autonomous mode and are otherwise disabled.

[0132] A video image synthesizer may include enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, if motion occurs in the video, noise reduction reduces the weight of information provided by adjacent frames and appropriately weights the spatial information. If the image or part of the image does not contain motion, the temporal noise reduction performed by the video image synthesizer can use information from previous images to reduce noise in the current image.

[0133] The video image synthesizer can also be configured to perform stereo rectification on the input stereo lens frame. Furthermore, the video image synthesizer can be used for user interface compositing when the operating system desktop is in use, eliminating the need for the GPU1108 to continuously render new faces. Even when the GPU1108 is powered on and actively performing 3D rendering, the video image synthesizer can be used to offload the GPU1108 to improve performance and responsiveness.

[0134] The SoC1104 may further include a Mobile Industry Processor Interface (MIPI) camera serial interface, a high-speed interface, and / or a video input block that can be used for camera and associated pixel input functions to receive video and input from a camera. The SoC1104 may further include an input / output controller that can be controlled by software and can be used to receive I / O signals that are not committed to a specific role.

[0135] The SoC1104 may further include a wide range of peripheral interfaces to enable communication with peripheral devices, audio codecs, power management, and / or other devices. The SoC1104 may be used to process data from cameras (connected via, for example, Gigabit Multimedia Serial Link and Ethernet®), sensors (e.g., LiDAR sensor 1164, RADAR sensor 1160, etc., which may be connected via Ethernet®), data from bus 1102 (e.g., vehicle speed, steering wheel position, etc.), and data from GNSS sensor 1158 (connected via, for example, Ethernet® or CAN bus). The SoC1104 may further include a dedicated high-performance mass storage controller which may include its own DMA engine and which may be used to free up CPU 1106 from routine data management tasks.

[0136] The SoC1104 may also be an inter-terminal platform with a flexible architecture that extends to automation levels 3-5, thereby providing a comprehensive functional safety architecture that leverages and efficiently uses computer vision and ADAS techniques for diversity and redundancy, and provides a platform for a flexible, reliable driving software stack along with deep learning tools. The SoC1104 can be faster, more reliable, more energy-efficient, and more space-efficient than conventional systems. For example, when the accelerator 1114 is coupled with the CPU 1106, the GPU 1108, and the data store 1116 can provide a fast and efficient platform for autonomous vehicles at levels 3-5.

[0137] Therefore, this technology brings capabilities and functionality that cannot be achieved by conventional systems. For example, computer vision algorithms can be executed on a CPU, which can be configured using high-level programming languages ​​such as the C programming language to execute a wide variety of processing algorithms across a wide variety of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. Specifically, many CPUs cannot execute real-time complex object detection algorithms, which are required for in-vehicle ADAS applications and actual Level 3-5 autonomous vehicles.

[0138] In contrast to conventional systems, by providing CPU complexes, GPU complexes, and hardware acceleration clusters, the technologies described herein enable multiple neural networks to run simultaneously and / or sequentially, and the results to be combined to enable Level 3–5 autonomous driving capabilities. For example, a DLA or a CNN running on a dGPU (e.g., GPU1120) may include text and word recognition, enabling a supercomputer to read and understand traffic signs, including signs for which the neural network has not been specifically trained. The DLA may further include a neural network capable of identifying, interpreting, and providing a semantic understanding of signs and passing that semantic understanding to a route planning module running on the CPU complex.

[0139] As another example, multiple neural networks may run simultaneously, as required for Level 3, 4, or 5 driving. For instance, a warning sign consisting of a flashing light and the text "Caution: Flashing light indicates frozen conditions" may be interpreted independently or collectively by several neural networks. The sign itself may be identified as a traffic sign by a first deployed neural network (e.g., a trained neural network), and the text "Flashing light indicates frozen conditions" may be interpreted by a second deployed neural network, informing the vehicle's route planning software (preferably running on a CPU complex) that frozen conditions are present when flashing light is detected. The flashing light may be identified by informing the vehicle's route planning software of the presence (or absence) of the flashing light, and by operating a third deployed neural network through multiple frames. All three neural networks can run simultaneously within the DLA and / or on the GPU1108, for example.

[0140] In some applications, a CNN for facial recognition and vehicle owner identification can use data from camera sensors to identify the presence of the legitimate driver and / or owner of vehicle 1100. An always-on sensor processing engine may be used to unlock the vehicle and turn on the lights when the owner approaches the driver's side door, and, in security mode, to stop the vehicle when the owner leaves the vehicle. In this way, SoC1104 provides security against theft and / or vehicle hijacking.

[0141] In another example, a CNN for emergency vehicle detection and identification can use data from microphone 1196 to detect and identify emergency vehicle sirens. In contrast to conventional systems that use a general classifier to detect sirens and manually extract features, SoC 1104 uses a CNN for classifying environmental and urban sounds, as well as for classifying visual data. In one preferred embodiment, a CNN running on DLA is trained to identify the relative terminal velocity of emergency vehicles (for example, by using the Doppler effect). The CNN may also be trained to identify emergency vehicles specific to the local area in which the vehicle is operating, as identified by GNSS sensor 1158. Thus, for example, when operating in Europe, the CNN would attempt to detect European sirens, and when in the United States, the CNN would attempt to identify only North American sirens. After an emergency vehicle is detected, a control program may be used, with the assistance of ultrasonic sensor 1162, to perform emergency vehicle safety routines such as slowing down the vehicle, stopping it at the side of the road, parking the vehicle, and / or idling the vehicle until the emergency vehicle has passed.

[0142] The vehicle may include a CPU 1118 (e.g., a discrete CPU, or dCPU) which can be connected to the SoC 1104 via a high-speed interconnect (e.g., PCIe). The CPU 1118 may include, for example, an x86 processor. The CPU 1118 may be used to perform any of a variety of functions, including, for example, mediating the consequences of a potential mismatch between ADAS sensors and the SoC 1104, and / or monitoring the status and condition of the controller 1136 and / or the infotainment SoC 1130.

[0143] Vehicle 1100 may include a GPU 1120 (e.g., a discrete GPU, or dGPU) which can be connected to SoC 1104 via a high-speed interconnect (e.g., NVIDIA NVLINK). The GPU 1120 can provide additional artificial intelligence capabilities, such as by running redundant and / or different neural networks, and may be used to train and / or update neural networks based on input from sensors in vehicle 1100 (e.g., sensor data).

[0144] Vehicle 1100 may further include a network interface 1124 which may include one or more wireless antennas 1126 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas and Bluetooth® antennas). The network interface 1124 may be used to enable wireless connectivity to a cloud over the Internet (e.g., with a server 1178 and / or other network devices), to other vehicles, and / or to computing devices (e.g., passenger client devices). To communicate with other vehicles, a direct link may be established between two vehicles, and / or an indirect link may be established (e.g., over a network and over the Internet). The direct link may be provided using a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link can provide information about vehicle 1100 regarding vehicles in close proximity to vehicle 1100 (e.g., vehicles in front of, beside, and / or behind vehicle 1100). This functionality may also be part of the vehicle 1100's cooperative adaptive cruise control function.

[0145] The network interface 1124 may include an SoC that provides modulation and demodulation functions and enables the controller 1136 to communicate over a wireless network. The network interface 1124 may include a radio frequency front end for upconversion from baseband to radio frequency and downconversion from radio frequency to baseband. Frequency conversion can be performed through well-known processes and / or using a superheterodyne process. In some examples, the radio frequency front end functionality may be provided by a separate chip. The network interface may include wireless functionality for communication over LTE, WCDMA®, UMTS, GSM, CDMA2000, Bluetooth®, Bluetooth® LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0146] The vehicle 1100 may further include a data store 1128 which may include storage outside the chip (for example, outside the SoC 1104). The data store 1128 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash, hard disk, and / or other components and / or devices capable of storing at least one bit of data.

[0147] The vehicle 1100 may further include a GNSS sensor 1158. The GNSS sensor 1158 (e.g., GPS, an assisted GPS sensor, a differential GPS (DGPS) sensor, etc.) assists in mapping, perception, occupy grid generation, and / or route planning functions. Any number of GNSS sensors 1158 may be used, including, but not limited to, a GPS using a USB connector with Ethernet® to a serial (RS-232) bridge.

[0148] Vehicle 1100 may further include a RADAR sensor 1160. The RADAR sensor 1160 may be used by vehicle 1100 for long-range vehicle detection, even in darkness and / or severe weather conditions. The RADAR functional safety level may be ASIL B. In some examples, the RADAR sensor 1160 may use CAN and / or bus 1102 for control and to access object tracking data (for example, to transmit data generated by the RADAR sensor 1160) using Ethernet® access for accessing raw data. A wide variety of RADAR sensor types may be used. For example, the RADAR sensor 1160 may be suitable for front, rear, and side RADAR use. In some examples, a pulsed Doppler RADAR sensor may be used.

[0149] The RADAR sensor 1160 may include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, and short-range side coverage. In some examples, the long-range RADAR may be used for adaptive cruise control functions. The long-range RADAR system can provide a wide field of view achieved by two or more independent scans, such as within a range of 250m. The RADAR sensor 1160 can help distinguish between static and moving objects and may be used by ADAS systems for emergency brake assist and forward collision warning. The long-range RADAR sensor may include a monostatic multimodal RADAR with multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In one example with six antennas, the four central antennas may create a focused beam pattern designed to record the area around the vehicle 1100 at high speed with minimal interference from traffic in adjacent lanes. The other two antennas can widen the field of view, enabling rapid detection of vehicles entering or leaving the lane of the vehicle 1100.

[0150] As an example, a medium-range RADAR system may include a range of up to 1160m (front) or 80m (rear) and a field of view of up to 42 degrees (front) or 1150 degrees (rear). A short-range RADAR system may include, but is not limited to, RADAR sensors designed to be mounted on both ends of the rear bumper. When mounted on both ends of the rear bumper, such a RADAR sensor system can create two beams that constantly monitor the blind spots behind and beside the vehicle.

[0151] Short-range radar systems can be used in ADAS systems for blind spot detection and / or lane change assistance.

[0152] The vehicle 1100 may further include ultrasonic sensors 1162. Positioned on the front, rear, and / or sides of the vehicle 1100, the ultrasonic sensors 1162 may be used for parking assistance and / or for creating and updating the occupancy grid. A wide variety of ultrasonic sensors 1162 may be used, and different ultrasonic sensors 1162 may be used for detection of different ranges (e.g., 2.5m, 4m). The ultrasonic sensors 1162 may operate at a functional safety level of ASIL B.

[0153] The vehicle 1100 may include a LiDAR sensor 1164. The LiDAR sensor 1164 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LiDAR sensor 1164 may also have a functional safety level of ASIL B. In some examples, the vehicle 1100 may include multiple LiDAR sensors 1164 (e.g., two, four, six, etc.) that can use Ethernet® (for example, to provide data to a Gigabit Ethernet® switch).

[0154] In some examples, the LIDAR sensor 1164 may have the ability to provide a list of objects and their distances within a 360-degree field of view. A commercially available LIDAR sensor 1164 may have an advertised range of approximately 1100m, for example, with an accuracy of 2cm to 3cm and support for 1100Mbps Ethernet® connectivity. In some examples, one or more non-protruding LIDAR sensors 1164 may be used. In such examples, the LIDAR sensor 1164 may be implemented as a small device that can be incorporated into the front, rear, side, and / or corners of a vehicle 1100. In such examples, the LIDAR sensor 1164 may have a range of 200m even for low-reflection objects and can provide a field of view up to 120 degrees horizontal and 35 degrees vertical. A front-mounted LIDAR sensor 1164 may be configured for a horizontal field of view between 45 and 135 degrees.

[0155] In some applications, LiDAR technologies such as 3D flash LiDAR may also be used. 3D flash LiDAR uses a laser flash as a source to illuminate the area around the vehicle up to approximately 200m. The flash LiDAR unit includes receptors that record the laser pulse travel time and reflected light on each pixel, sequentially corresponding to the range from the vehicle to the object. Flash LiDAR can enable the generation of high-precision and distortion-free images of the surroundings with every laser flash. In some applications, four flash LiDAR sensors may be deployed, one on each side of the vehicle 1100. Available 3D flash LiDAR systems include solid-state 3D steering array LiDAR cameras (e.g., non-scanning LiDAR devices) that have no moving parts other than a blower. The flash LiDAR device can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture reflected laser light in the form of a 3D range point cloud and co-documented intensity data. By using flash LiDAR, and because flash LiDAR is a solid-state device with no moving parts, the LiDAR sensor 1164 may be less susceptible to motion blur, vibration, and / or shock.

[0156] The vehicle may further include an IMU sensor 1166. In some examples, the IMU sensor 1166 may be positioned in the center of the rear axle of the vehicle 1100. The IMU sensor 1166 may include, but is not limited to, an accelerometer, magnetometer, gyroscope, magnetic compass, and / or other sensor types. In some examples, such as in a 6-axis application, the IMU sensor 1166 may include an accelerometer and a gyroscope, while in a 9-axis application, the IMU sensor 1166 may include an accelerometer, a gyroscope, and a magnetometer.

[0157] In some embodiments, the IMU sensor 1166 may be implemented as a compact, high-performance GPS-aided inertial navigation system (GPS / INS) that combines a micro-electro-mechanical system (MEMS) inertial sensor, a high-sensitivity GPS receiver, and an advanced Kalman filtering algorithm to provide position, velocity, and attitude estimates. As such, in some examples, the IMU sensor 1166 may enable the vehicle 1100 to estimate its direction of travel without requiring input from a magnetic sensor by directly observing and correlating changes in velocity from the GPS to the IMU sensor 1166. In some examples, the IMU sensor 1166 and the GNSS sensor 1158 may be combined in a single integrated unit.

[0158] The vehicle may include a microphone 1196 placed inside and / or around the vehicle 1100. The microphone 1196 may, among other things, be used for emergency vehicle detection and identification.

[0159] The vehicle may further include any number of camera types, including a stereo camera 1168, a wide-view camera 1170, an infrared camera 1172, a surround camera 1174, a long-range and / or medium-range camera 1198, and / or other camera types. The cameras may be used to capture 360-degree image data of the vehicle 1100. The type of camera used will depend on the embodiment and requirements of the vehicle 1100, and any combination of camera types may be used to achieve the required coverage around the vehicle 1100. In addition, the number of cameras may vary depending on the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. The cameras may, as an example, support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet®. Each camera is described in more detail herein in relation to Figures 11A and 11B.

[0160] The vehicle 1100 may further include a vibration sensor 1142. The vibration sensor 1142 can measure vibrations of vehicle components, such as axles. For example, a change in vibration may indicate a change in the road surface. In another example, when two or more vibration sensors 1142 are used, the difference in vibration may be used to determine friction or slippage of the road surface (for example, when the difference in vibration is between a power-driven axle and a free-rotating axle).

[0161] Vehicle 1100 may include an ADAS system 1138. In some examples, the ADAS system 1138 may include a System of Control (SoC). The ADAS system 1138 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward crash warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keep assist (LKA), blind spot warning (BSW), rear cross-traffic warning (RCTW), collision warning system (CWS), lane centering (LC), and / or other features and functions.

[0162] The ACC system may utilize a radar sensor 1160, a lithium-ion sensor 1164, and / or a camera. The ACC system may include longitudinal ACC and / or lateral ACC. Longitudinal ACC monitors and controls the distance to the vehicle immediately in front of vehicle 1100 and automatically adjusts the vehicle speed to maintain a safe distance from the vehicle ahead. Lateral ACC performs distance maintenance and advises vehicle 1100 to change lanes when necessary. Lateral ACC is related to other ADAS applications such as LCA and CWS.

[0163] CACC uses information from other vehicles that can be received from other vehicles via a wireless link through a network interface 1124 and / or a wireless antenna 1126, or indirectly via a network connection (e.g., via the Internet). Direct links may be provided by vehicle-to-vehicle (V2V) communication links, while indirect links may be infrastructure-to-vehicle (I2V) communication links. Generally, the V2V communication concept provides information about the vehicle immediately ahead (e.g., a vehicle in the same lane as vehicle 1100, immediately in front of vehicle 1100), while the I2V communication concept provides information about traffic further ahead. A CACC system may include either or both I2V and V2V information sources. Given information about vehicles ahead of vehicle 1100, CACC can be more reliable, and CACC has the potential to make traffic flow smoother and reduce road congestion.

[0164] The FCW system is designed to warn the driver of hazards so that the driver can take corrective action. The FCW system uses a forward-facing camera and / or radar sensor 1160, coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibration components. The FCW system can provide warnings in the form of audible, visual, vibration, and / or quick brake pulses.

[0165] An AEB system can detect an imminent forward collision with another vehicle or object and automatically apply the brakes if the driver does not take corrective action within a specified time or distance parameter. The AEB system may use a forward-facing camera and / or radar sensor 1160 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. When the AEB system detects a hazard, it typically first warns the driver to take corrective action to avoid the collision. If the driver does not take corrective action, the AEB system may automatically apply the brakes as part of an effort to prevent, or at least mitigate, the impact of the anticipated collision. The AEB system may include techniques such as dynamic brake support and / or collision emergency braking.

[0166] The LDW system warns the driver when the vehicle 1100 crosses a lane marking by providing visual, audible, and / or tactile warnings, such as vibration of the steering wheel or seat. The LDW system does not activate when the driver indicates an intentional lane departure by activating the turn signal. The LDW system may use a forward-facing camera connected to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback, such as a display, speaker, and / or vibration components.

[0167] The LKA system is a modified form of the LDW system. The LKA system provides steering input or braking to correct the vehicle 1100 when the vehicle 1100 begins to drift out of its lane.

[0168] The BSW system detects and warns the driver of a vehicle in the vehicle's blind spot. The BSW system can provide visual, audible, and / or tactile warnings to indicate that merging or changing lanes is unsafe. The system can provide additional warnings when the driver uses the turn signal. The BSW system can use a rear-facing camera and / or radar sensor 1160 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback, such as a display, speaker, and / or vibration component.

[0169] The RCTW system can provide visual, audible, and / or haptic notifications when an object is detected outside the range of the rear camera while the vehicle 1100 is reversing. Some RCTW systems include AEB to ensure that the vehicle brakes are applied to avoid a collision. The RCTW system may use one or more rear-facing RADAR sensors 1160 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which are electrically coupled to driver feedback, such as a display, speaker, and / or vibration component.

[0170] Conventional ADAS systems warn the driver, allowing the driver to determine whether a safe condition truly exists and act accordingly. However, conventional ADAS systems have a tendency to produce misjudgments that are usually not catastrophic but can frustrate and distract the driver. In the autonomous vehicle 1100, however, if the results are contradictory, the vehicle 1100 itself must decide whether to heed the results from the primary computer or the secondary computer (e.g., the first controller 1136 or the second controller 1136). For example, in some embodiments, the ADAS system 1138 may also be a backup and / or secondary computer for providing perceptual information to a backup computer rationality module. The backup computer rationality monitor can run redundant and diverse software on hardware components to detect failures in perceptual and dynamic driving tasks. The output from the ADAS system 1138 may be provided to the supervisory MCU. If the outputs from the primary and secondary computers are contradictory, the supervisory MCU must decide how to reconcile the contradiction to ensure safe operation.

[0171] In some implementations, the primary computer may be configured to provide the supervising MCU with a reliability score indicating the reliability of the primary computer in the selected outcome. If the reliability score exceeds a threshold, the supervising MCU may follow the primary computer's instructions, regardless of whether the secondary computer gives conflicting or inconsistent results. If the reliability score does not meet the threshold, and if the primary and secondary computers produce different results (e.g., conflicting results), the supervising MCU may mediate between the computers to determine an appropriate outcome.

[0172] The supervisory MCU may be configured to run a neural network trained and configured to determine, based on the outputs from the primary and secondary computers, when a secondary computer provides a false alarm. Thus, the neural network in the supervisory MCU can learn when the output of the secondary computer is reliable and when it is not. For example, when the secondary computer is a radar-based forward crossing (FCW) system, the neural network in the supervisory MCU can learn when the FCW identifies a metal object that is not actually dangerous, such as a sewer grate or manhole cover, which triggers an alarm. Similarly, when the secondary computer is a camera-based lane departure warning (LDW) system, the neural network in the supervisory MCU can learn to ignore the LDW when a cyclist or pedestrian is present and lane departure is actually the safest operation. In embodiments including a neural network running on the supervisory MCU, the supervisory MCU may include at least one DLA or GPU suitable for running a neural network with associated memory. In a preferred embodiment, the supervisory MCU may comprise and / or be included as a component of the SoC1104.

[0173] In other examples, the ADAS system 1138 may include a secondary computer that performs ADAS functions using conventional rules of computer vision. As such, the secondary computer may use classical computer vision rules (if-then), and the presence of a neural network within the supervisory MCU can improve reliability, safety, and performance. For example, diverse implementations and intentional non-identities make the entire system more fault-tolerant, particularly to failures caused by software (or software-hardware interface) functions. For instance, if a software bug or error exists in the software running on the primary computer, and non-identical software code running on the secondary computer produces the same overall result, the supervisory MCU may have greater confidence that the overall result is correct and that the bug in the software or hardware on the primary computer did not cause a critical error.

[0174] In some examples, the output of the ADAS system 1138 may be supplied to the perception block and / or the dynamic driving task block of the primary computer. For example, if the ADAS system 1138 indicates a forward collision warning due to an object immediately ahead, the perception block can use this information when identifying the object. In other examples, the secondary computer may have its own neural network, which is trained as described herein and therefore reduces the risk of misjudgment.

[0175] Vehicle 1100 may further include an infotainment SoC 1130 (for example, an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system does not have to be an SoC and may include two or more discrete components. The infotainment SoC 1130 may include a combination of hardware and software that can be used to provide vehicle 1100 with audio (e.g., music, personal digital assistant, navigation commands, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and / or information services (e.g., a navigation system, rear parking assist, radio data system, fuel level, total mileage, brake fuel level, oil level, door open / close, air filter information, and other vehicle-related information). For example, the infotainment SoC 1130 may also include wireless, disc player, navigation system, video player, USB and Bluetooth® connectivity, car computer, in-car entertainment, Wi-Fi, steering wheel audio control unit, hands-free voice control, heads-up display (HUD), HMI display 1134, telematics device, control panel (for example, for controlling and / or interacting with various components, features, and / or systems), and / or other components. The infotainment SoC 1130 may be further used to provide information (for example, visual and / or audible) to the vehicle user, such as information from the ADAS system 1138, autonomous driving information such as planned vehicle operation, trajectory, surrounding environment information (for example, intersection information, vehicle information, road information, etc.), and / or other information.

[0176] The infotainment SoC 1130 may include GPU functionality. The infotainment SoC 1130 can communicate with other devices, systems, and / or components of the vehicle 1100 via a bus 1102 (e.g., CAN bus, Ethernet®, etc.). In some examples, the infotainment SoC 1130 may be coupled to a supervisory MCU so that the infotainment system's GPU can perform certain self-drive functions in the event of a primary controller 1136 (e.g., the vehicle 1100's primary and / or backup computer) failure. In such examples, the infotainment SoC 1130 can put the vehicle 1100 into a driver-assistant mode for safe stopping, as described herein.

[0177] Vehicle 1100 may further include an instrument cluster 1132 (e.g., a digital dash, an electronic instrument cluster, a digital instrument panel, etc.). The instrument cluster 1132 may include a controller and / or a supercomputer (e.g., a discrete controller or supercomputer). The instrument cluster 1132 may include a set of instruments such as a speedometer, fuel level indicator, oil pressure indicator, tachometer, odometer, turn signals, gear shift position indicator, seat belt warning light, parking brake warning light, engine fault light, airbag (SRS) system information, lighting control device, safety system control device, and navigation information. In some examples, information may be displayed and / or shared between the infotainment SoC 1130 and the instrument cluster 1132. In other words, the instrument cluster 1132 may be included as part of the infotainment SoC 1130, and vice versa.

[0178] Figure 11D is a system diagram of communication between the cloud-based server of Figure 11A and an exemplary autonomous vehicle 1100 according to several embodiments of the present disclosure. System 1176 may include a server 1178, a network 1190, and a vehicle including the vehicle 1100. Server 1178 may include a plurality of GPUs 1184(A) to 1184(H) (collectively referred to herein as GPU 1184), PCIe switches 1182(A) to 1182(H) (collectively referred to herein as PCIe switch 1182), and / or CPUs 1180(A) to 1180(B) (collectively referred to herein as CPU 1180). The GPUs 1184, CPUs 1180, and PCIe switches may be interconnected by high-speed interconnects, such as, for example, NVLink interfaces 1188 and / or PCIe connections 1186 developed by NVIDIA. In some examples, the GPU 1184 is connected via NVLink and / or NVSwitch SoCs, and the GPU 1184 and PCIe switch 1182 are connected via PCIe interconnects. Eight GPU 1184s, two CPU 1180s, and two PCIe switches are illustrated, but this is not intended to be limiting. Depending on the embodiment, each server 1178 may include any number of GPU 1184s, CPU 1180s, and / or PCIe switches. For example, server 1178 may include eight, sixteen, thirty-two, and / or more GPU 1184s, respectively.

[0179] Server 1178 can receive image data from vehicles via network 1190, representing images showing unexpected or altered road conditions, such as recently started road construction. Server 1178 can transmit to vehicles via network 1190, the neural network 1192, and / or map information 1194, which includes information about traffic and road conditions. Updates to map information 1194 may include updates to HD map 1122, such as information about construction sites, potholes, detours, floods, and / or other obstacles. In some examples, the neural network 1192, the updated neural network 1192, and / or map information 1194 may have arisen from new training and / or experience represented in data received from any number of vehicles in the environment, and / or based on training performed in a data center (for example, using server 1178 and / or other servers).

[0180] Server 1178 may be used to train a machine learning model (e.g., a neural network) based on training data. The training data may be generated by a vehicle and / or in a simulation (e.g., using a game engine). In some instances, the training data is tagged (e.g., if the neural network benefits from supervised learning) and / or otherwise pre-processed, while in other instances, the training data is not tagged and / or pre-processed (e.g., if the neural network does not require supervised learning). Training may be performed according to any one or more classes of machine learning techniques, including but not limited to the following: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, associative learning, transfer learning, feature learning (including key component and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including pre-dictionary learning), rule-based machine learning, anomaly detection, and variations or combinations thereof. After the machine learning model has been trained, it may be used by the vehicle (for example, transmitted to the vehicle via network 1190), and / or the machine learning model may be used by server 1178 to remotely monitor the vehicle.

[0181] In some examples, Server 1178 can receive data from vehicles and apply it to a state-of-the-art real-time neural network for real-time intelligent inference. Server 1178 may include a deep learning supercomputer and / or dedicated AI computer powered by a GPU 1184, such as the DGX and DGX Station Machine developed by NVIDIA. However, in some examples, Server 1178 may include a deep learning infrastructure that uses only a CPU-powered data center.

[0182] The deep learning infrastructure of server 1178 can have the capability for high-speed real-time inference, which can be used to evaluate and verify the condition of the processor, software, and / or associated hardware within vehicle 1100. For example, the deep learning infrastructure can receive periodic updates from vehicle 1100, such as images of a sequence and / or objects located within images of that sequence (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure can run its own neural network to identify objects and compare them with objects identified by vehicle 1100, and if the results do not match and the infrastructure concludes that the AI ​​within vehicle 1100 is not functioning properly, server 1178 can send a signal to vehicle 1100 instructing the vehicle's fail-safe computer to infer control, notify passengers, and complete a safe parking operation.

[0183] For inference, server 1178 may include GPU 1184 and one or more programmable inference accelerators (e.g., NVIDIA TensorRT). The combination of a GPU-powered server and inference accelerator can enable real-time responsiveness. In other examples, such as when performance is not a major requirement, a server powered by a CPU, FPGA, and other processors may be used for inference.

[0184] (Example computing device) Figure 12 is a block diagram of an example of a computing device 1200 suitable for use in implementing some embodiments of the present disclosure. The computing device 1200 may include an interconnection system 1202 that directly or indirectly connects the following devices: memory 1204, one or more central processing units (CPUs) 1206, one or more graphics processing units (GPUs) 1208, a communication interface 1210, input / output (I / O) ports 1212, input / output components 1214, a power supply device 1216, one or more presentation components 1218 (e.g., a display), and one or more logical units 1220. In at least one embodiment, the computing device 1200 may include one or more virtual machines (VMs), and / or any of its components may include virtual components (e.g., virtual hardware components). As an unrestricted example, one or more of the GPUs 1208 may include one or more vGPUs, one or more of the CPUs 1206 may include one or more vCPUs, and / or one or more of the logical units 1220 may include one or more virtual logical units. As such, the computing device 1200 may include discrete components (e.g., an entire GPU dedicated to the computing device 1200), virtual components (e.g., a portion of a GPU dedicated to the computing device 1200), or a combination thereof.

[0185] The various blocks in Figure 12 are shown connected by lines via the interconnection system 1202, but this is not intended to be limiting and is simply for clarity. For example, in some embodiments, a presentation component 1218, such as a display device, could be considered an I / O component 1214 (for example, if the display is a touchscreen). As another example, the CPU 1206 and / or GPU 1208 could include memory (for example, memory 1204 could represent a storage device in addition to the memory of the GPU 1208, CPU 1206, and / or other components). In other words, the computing devices in Figure 12 are merely illustrative. Categories such as “workstation,” “server,” “laptop,” “desktop,” “tablet,” “client device,” “mobile device,” “handheld device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types are all intended to fall within the scope of the computing devices in Figure 12 and are therefore not distinguished.

[0186] The interconnection system 1202 may represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnection system 1202 may include one or more bus or link types, such as an industry standard architecture (ISA) bus, an extended industry standard architecture (EISA) bus, a VESA (video electronics standards association) bus, a peripheral component interconnect (PCI) bus, a peripheral component interconnect express (PCIe) bus, and / or other types of buses or links. In some embodiments, direct connections exist between components. For example, the CPU 1206 may be directly connected to the memory 1204. Furthermore, the CPU 1206 may be directly connected to the GPU 1208. Where direct or point-to-point connections exist between components, the interconnection system 1202 may include PCIe links for implementing the connections. In these examples, the PCI bus does not need to be included in the computing device 1200.

[0187] Memory 1204 may include any of various computer-readable media. The computer-readable media may be any available media accessible by the computing device 1200. The computer-readable media may include both volatile and non-volatile media, as well as removable and non-removable media. By example, and not by limitation, the computer-readable media may include computer storage media and communication media.

[0188] Computer storage media may include both volatile and non-volatile media, and / or removable and non-removable media, implemented in any method or technique for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 1204 can store computer-readable instructions (e.g., representing programs and / or program elements), such as an operating system. Computer storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory, or other memory technologies, CD-ROM, digital versatile disk (DVD), or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage, or other magnetic storage devices, or any other media that can be used to store desired information and can be accessed by computing device 1200. In this specification, computer storage media does not include signals themselves.

[0189] Computer storage media include any information distribution medium that can implement computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals such as carrier waves or other transfer mechanisms. The term “modulated data signal” may refer to a signal that has been modified in a manner that has one or more of its characteristic sets or encodes information within the signal. By example, and not by limitation, computer storage media may include wired media such as wired networks or direct wired connections, and wireless media such as acoustic, RF, infrared, and other wireless media. Any combination of the foregoing should also be included in the scope of computer-readable media.

[0190] The CPU 1206 may be configured to execute at least some computer-readable instructions to control one or more components of the computing device 1200 to execute one or more of the methods and / or processes described herein. The CPU 1206 may include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) each capable of processing a large number of software threads concurrently. The CPU 1206 may include any type of processor, and depending on the type of computing device 1200 in which it is implemented, it may include different types of processors (e.g., a processor with fewer cores for mobile devices and a processor with more cores for servers). For example, depending on the type of computing device 1200, the processor may be an Advanced RISC Machines (ARM) processor implemented using Reduced Instruction Set Computing (RISC), or an x86 processor implemented using Complex Instruction Set Computing (CISC). The computing device 1200 may include one or more CPUs 1206 in addition to one or more microprocessors or auxiliary coprocessors, such as a computing coprocessor.

[0191] In addition to or instead of the CPU 1206, the GPU 1208 may be configured to execute at least some computer-readable instructions to control one or more components of the computing device 1200 to execute one or more of the methods and / or processes described herein. One or more of the GPUs 1208 may be an integrated GPU (for example, together with one or more of the CPUs 1206), and / or one or more of the GPUs 1208 may be a discrete GPU. In embodiments, one or more of the GPUs 1208 may be a coprocessor of one or more of the CPUs 1206. The GPU 1208 may be used by the computing device 1200 to render graphics (for example, 3D graphics) or to perform general-purpose computing. For example, the GPU 1208 may be used for general-purpose computing on a GPU (GPGPU). The GPU 1208 may include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. GPU1208 can generate pixel data for an output image in response to rendering commands (for example, rendering commands from CPU1206 received via the host interface). GPU1208 may include graphics memory, such as display memory, for storing pixel data or any other appropriate data, such as GPGPU data. Display memory may be included as part of memory 1204. GPU1208 may include two or more GPUs operating in parallel (for example, via a link). The link can connect directly to the GPUs (for example, using NVLINK) or connect the GPUs via a switch (for example, using NVSwitch). When coupled together, each GPU1208 can generate pixel data or GPGPU data for different parts of an output or different outputs (for example, the first GPU for the first image and the second GPU for the second image). Each GPU may have its own memory or may share memory with other GPUs.

[0192] In addition to or instead of the CPU 1206 and / or GPU 1208, the logic unit 1220 may be configured to execute at least some computer-readable instructions to control one or more of the computing devices 1200 to execute one or more of the methods and / or processes described herein. In embodiments, the CPU 1206, GPU 1208, and / or the logic unit 1220 can execute any combination of methods, processes, and / or parts thereof discretely or congruently. One or more of the logic units 1220 may be part of and / or integrated with one or more of the CPU 1206 and / or GPU 1208, and / or one or more of the logic units 1220 may be discrete components of the CPU 1206 and / or GPU 1208 or otherwise external to them. In embodiments, one or more of the logic units 1220 may be coprocessors of one or more of the CPU 1206 and / or one or more of the GPU 1208.

[0193] Examples of logical unit 1220 include one or more processing cores and / or components thereof, such as a Data Processing Unit (DPU), Tensor Core (TC), Tensor Processing Unit (TPU), Pixel Visual Core (PVC), Vision Processing Unit (VPU), Graphics Processing Cluster (GPC), Texture Processing Cluster (TPC), Streaming Multiprocessor (SM), Tree Traversal Unit (TTU), Artificial Intelligence Accelerator (AIA), and Deep Learning Accelerator (DLA). This includes an Accelerator, a Logical Unit (ALU), an Application-Specific Integrated Circuit (ASIC), a Floating-Point Unit (FPU), input / output (I / O) elements, a Peripheral Component Interconnect (PCI) or Peripheral Component Interconnect Express (PCIe) element, and / or similar.

[0194] The communication interface 1210 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 1200 to communicate with other computing devices via an electronic communication network, including wired and / or wireless communication. The communication interface 1210 may include components and functions to enable communication over any of several different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth®, Bluetooth® LE, ZigBee, etc.), wired networks (e.g., communicating via Ethernet® or InfiniBand), low-power wide-area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, the logic unit 1220 and / or the communication interface 1210 may include one or more data processing units (DPUs) to transmit data received via the network and / or through the interconnection system 1202 directly to one or more GPUs 1208 (e.g., their memory).

[0195] I / O port 1212 can enable the computing device 1200 to be logically connected to other devices, including I / O components 1214, presentation components 1218, and / or other components, some of which can be built into (e.g., integrated into) the computing device 1200. Exemplary I / O components 1214 include microphones, mice, keyboards, joysticks, gamepads, game controllers, satellite dishes, scanners, printers, wireless devices, etc. I / O components 1214 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by the user. In some cases, the input may be transmitted to appropriate network elements for further processing. The NUI may implement any combination of speech recognition, stylus recognition, facial recognition, biometric recognition, on-screen and beside-screen gesture recognition, air gestures, head and target tracking, and touch recognition related to the display of the computing device 1200 (as described in more detail below). The computing device 1200 may include depth cameras, such as stereoscope camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof, for gesture detection and recognition. Additionally, the computing device 1200 may include accelerometers or gyroscopes that enable motion detection (for example, as part of an inertia measurement unit (IMU)). In some examples, the output of the accelerometer or gyroscope may be used by the computing device 1200 to render immersive augmented reality or virtual reality.

[0196] The power supply device 1216 may include a hardwired power supply device, a battery power supply device, or a combination thereof. The power supply device 1216 can provide power to the computing device 1200 to enable the components of the computing device 1200 to operate.

[0197] The presentation component 1218 may include a display (e.g., a monitor, touch screen, television screen, head-up display device (HUD), other display types, or a combination thereof), a speaker, and / or other presentation components. The presentation component 1218 can receive data from other components (e.g., GPU 1208, CPU 1206, DPU, etc.) and output data (e.g., as images, videos, sounds, etc.).

[0198] (Example data center) Figure 13 shows an exemplary data center 1300 that may be used in at least one embodiment of the present disclosure. The data center 1300 may include a data center infrastructure layer 1310, a framework layer 1320, a software layer 1330, and / or an application layer 1340.

[0199] As shown in Figure 13, the data center infrastructure layer 1310 may include a resource orchestrator 1312, grouped computing resources 1314, and node computing resources ("node CRs") 1316(1) to 1316(N), where "N" represents any positive integer. In at least one embodiment, the node CRs 1316(1) to 1316(N) may include, but are not limited to, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field-programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output (NW I / O) devices, network switches, virtual machines (VMs), power modules, and / or cooling modules. In some embodiments, one or more nodes CR1316(1) to 1316(N) may correspond to a server having one or more of the aforementioned computing resources. In addition, in some embodiments, nodes CR1316(1) to 1316(N) may include one or more virtual components, such as vGPUs, vCPUs, and / or similar, and / or one or more nodes CR1316(1) to 1316(N) may correspond to a virtual machine (VM).

[0200] In at least one embodiment, the grouped computing resources 1314 may include a separate group of nodes CR1316 housed in one or more racks (not shown), or a number of racks housed in data centers in various geographical locations (also not shown). The separate group of nodes CR1316 within the grouped computing resources 1314 may include grouped compute, network, memory, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, several nodes CR1316, including CPUs, GPUs, DPUs, and / or other processors, may be grouped in one or more racks to provide computing resources to support one or more workloads. The one or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0201] The resource orchestrator 1312 can configure or otherwise control one or more nodes CR1316(1) to 1316(N) and / or grouped computing resources 1314. In at least one embodiment, the resource orchestrator 1312 may include a software design infrastructure ("SDI") management entity for the data center 1300. The resource orchestrator 1312 may include hardware, software, or any combination thereof.

[0202] In at least one embodiment, as shown in Figure 13, the framework layer 1320 may include a job scheduler 1333, a configuration manager 1334, a resource manager 1336, and / or a distributed file system 1338. The framework layer 1320 may include a framework to support the software 1332 of the software layer 1330 and / or one or more applications 1342 of the application layer 1340. The software 1332 or applications 1342 may include web-based service software or applications, such as those provided by Amazon Web Services®, Google Cloud, and Microsoft Azure, respectively. The framework layer 1320 may be, but is not limited to, a type of free and open-source software web application framework, such as Apache Spark® ("Spark"), which may use the distributed file system 1338 for large-scale data processing (e.g., "big data"). In at least one embodiment, the job scheduler 1333 may include a Spark driver to facilitate scheduling of workloads supported by various layers of the data center 1300. The configuration manager 1334 may have the ability to configure different layers, for example, a software layer 1330 and a framework layer 1320 including Spark and a distributed file system 1338 to support large-scale data processing. The resource manager 1336 may have the ability to manage clustered or grouped computing resources mapped or allocated for support of the distributed file system 1338 and the job scheduler 1333. In at least one embodiment, the clustered or grouped computing resources may include computing resources 1314 grouped in the data center infrastructure layer 1310. The resource manager 1336 can coordinate with the resource orchestrator 1312 to manage these mapped or allocated computing resources.

[0203] In at least one embodiment, the software 1332 included in the software layer 1330 may include software used by at least a portion of nodes CR1316(1) to 1316(N), grouped computing resources 1314, and / or the distributed file system 1338 of the framework layer 1320. One or more types of software may include, but are not limited to, internet web page search software, email virus scanning software, database software, and streaming video content software.

[0204] In at least one embodiment, the application 1342 included in the application layer 1340 may include one or more types of applications used by at least a portion of the nodes CR1316(1) to 1316(N), the grouped computing resources 1314, and / or the distributed file system 1338 of the framework layer 1320. One or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments.

[0205] In at least one embodiment, any of the configuration manager 1334, resource manager 1336, and resource orchestrator 1312 may implement any number and type of self-rewriting actions based on any amount and type of data obtained in any technically possible manner. Self-rewriting actions may free the data center operator of data center 1300 from making potentially poor configuration decisions and possibly avoiding underutilized and / or underperforming parts of the data center.

[0206] The data center 1300 may include tools, services, software, or other resources for training one or more machine learning models or for predicting or inferring information using one or more machine learning models according to one or more embodiments described herein. For example, a machine learning model may be trained by calculating weight parameters by a neural network architecture using the software and / or computing resources described herein with respect to the data center 1300. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks may be used to infer or predict information using the resources described herein with respect to the data center 1300 by using weight parameters calculated via one or more training techniques, not limited to those described herein.

[0207] In at least one embodiment, the data center 1300 may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, and / or other hardware (or corresponding virtual computing resources) to perform training and / or inference using the aforementioned resources. Furthermore, one or more of the aforementioned software and / or hardware resources may be configured as services that enable a user to train or perform inference of information, such as image recognition, speech recognition, or other artificial intelligence services.

[0208] (Example network environment) A network environment suitable for use in implementing embodiments of the embodiments of this disclosure may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. Each client device, server, and / or other device type (for example, each device) may be implemented as one or more instances of the computing device 1200 in Figure 12, for example, each device may include similar components, features, and / or functionalities of the computing device 1200. In addition, if backend devices (for example, servers, NAS, etc.) are implemented, they may be included as part of the data center 1300, examples of which are further detailed herein with respect to Figure 13.

[0209] Components of a network environment may communicate with one another via the network, either wired, wirelessly, or both. A network may comprise multiple networks, or one of multiple networks. Examples include one or more wide-area networks (WANs), one or more local-area networks (LANs), one or more public networks, such as the Internet and / or the Public Switched Telephone Network (PSTN), and / or one or more private networks. If a network includes a wireless telecommunications network, its components, such as base stations, towers, or access points (and other components), may provide wireless connectivity.

[0210] Compatible network environments may include one or more peer-to-peer network environments (in which case servers may not be included in the network environment) and one or more client-server network environments (in which case one or more servers may be included in the network environment). In a peer-to-peer network environment, the functionality described herein with respect to the server can be implemented on any number of client devices.

[0211] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, or a combination thereof. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more of the servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework to support the software in the software layer and / or one or more applications in the application layer. The software or applications may each include web-based service software or applications. In embodiments, one or more of the client devices may use the web-based service software or applications (for example, by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a type of free and open-source software web application framework that may use a distributed file system for, for example, large-scale data processing (e.g., “big data”).

[0212] A cloud-based network environment may provide cloud computing and / or cloud storage that implements any combination of the computing and / or data storage functions (or one or more of them) described herein. Any of these various functions may be distributed across multiple locations from a central or core server (such as one or more data centers that may be distributed across states, territories, countries, or the world). If the connection to the user (e.g., a client device) is relatively close to the edge server, the core server may assign at least some of its functionality to the edge server. The cloud-based network environment may be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0213] A client device may include at least some of the components, features, and functionalities of the exemplary computing device 1200 described herein with respect to Figure 12. Not limited to, but as an example, a client device may be implemented as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, video camera, surveillance device or system, vehicle, boat, airship, virtual machine, drone, robot, handheld communication device, hospital device, gaming device or system, entertainment system, vehicle computer system, embedded system controller, remote control, instrument, consumer electronic device, workstation, edge device, any combination of these depicted devices, or any other suitable device.

[0214] This disclosure may be described in general terms with computer code or machine-available instructions, including computer-executable instructions such as program modules, which are executed by computers or other machines, such as personal digital assistants or other handheld devices. Generally, a program module, including routines, programs, objects, components, and data structures, refers to code that performs a specific task or implements a specific abstract data type. This disclosure may be implemented in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, and more specialized computing devices. This disclosure may also be implemented in a distributed computing environment where tasks are performed by remote processing devices linked over a communication network.

[0215] In this specification, any “and / or” statement relating to two or more elements should be interpreted as meaning only one element or a combination of elements. For example, “element A, element B, and / or element C” may include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. In addition, “at least one of element A or element B” may include at least one element A, at least one element B, or at least one element A and at least one element B. Furthermore, “at least one of element A and element B” may include at least one element A, at least one element B, or at least one element A and at least one element B.

[0216] The inventions of this disclosure are described in a manner that is specific in order to satisfy statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors intend that the inventions described in the claims may be carried out in other ways, including different steps or combinations of steps similar to those described herein, in conjunction with other current or future technologies. Furthermore, the terms “step” and / or “block” may be used herein to imply different elements of the way in which they are used, but these terms should not be construed as implying any particular order among the various steps disclosed herein unless the order of the individual steps is expressly stated and, when so, is explicitly stated.

Claims

1. A step of evaluating one or more attributes of one or more data samples applied to at least one machine learning model (MLM), A step of identifying at least one value of at least one of the one or more attributes based at least on the evaluation described above, The steps include applying at least one input to one or more generating MLMs to generate one or more additional data samples corresponding to the at least one value, The steps include training the at least one MLM using the one or more additional data samples and Includes, A method wherein the step of generating one or more additional data samples includes determining whether to generate one or more additional data samples by analyzing at least one performance metric across attributes, wherein the at least one performance metric corresponds to the inference accuracy of the at least one MLM for at least one of the one or more attributes.

2. The method according to claim 1, wherein the one or more generating MLMs include a synthetic generator that generates the one or more data samples as a synthesis of the one or more attributes.

3. The method according to claim 1, wherein the step of evaluating one or more attributes includes evaluating one or more attributes using at least one performance metric corresponding to the inference accuracy of at least one MLM for at least one of the one or more attributes, and the identifying step includes selecting at least one of the one or more attributes on at least the fact that the inference accuracy is below a threshold accuracy level.

4. The method according to claim 1, wherein the at least one value of the at least one attribute is one or more values ​​of a first set of attributes, and the evaluation step includes evaluating the one or more attributes using at least one performance metric corresponding to the inference accuracy of the at least one MLM of one or more values ​​of the first set for one or more values ​​of the second set of attributes.

5. The method according to claim 1, wherein the one or more data samples and the one or more additional data samples include images depicting one or more objects, and the one or more attributes are attributes of the one or more objects.

6. The method according to claim 1, wherein the evaluation step includes evaluating the one or more attributes using at least one performance metric corresponding to the inference accuracy of the at least one MLM of a pattern over time represented by one or more values ​​of the at least one attribute across a plurality of data samples, the at least one value of the at least one attribute includes the pattern over time.

7. The method according to claim 6, wherein the time-series pattern corresponds to one or more of the frequencies, amplitudes, speeds, or durations of one or more events represented using the values ​​of the at least one attribute.

8. The aforementioned at least one attribute is The person's age, ethnicity, hair length, head position, whether the person wears glasses, whether the person has a beard, emotions, blink rate, degree of eyelid opening, eye makeup, blink amplitude, blink duration, facial pattern, whether the person is wearing a mask, lighting conditions, facial expression, whether the person is emphasized in the one or more images, whether the person's background is emphasized in the one or more images, or whether the person's foreground is emphasized in the one or more images. The method according to claim 1, wherein one or more of the above are defined with respect to the person depicted in the one or more images.

9. The method according to claim 1, wherein the one or more generating MLMs include an unconditional generating model, and the at least one input derives the unconditional generating model when generating the one or more data samples using a first energy function defining the semantics of a first attribute of the at least one attribute and a second energy function defining the semantics of a second attribute of the at least one attribute.

10. A system for controllably generating one or more data samples of one or more attributes, To generate one or more performance metric values ​​corresponding to the inference accuracy of at least one machine learning model (MLM) for one or more attributes of one or more objects depicted in a first set of images, applied to at least one MLM; Using one or more of the performance metric values, determine that the inference accuracy of the at least one MLM is below one or more thresholds for at least one value of at least one of the attributes of the one or more attributes, Applying at least one input to one or more generating MLMs to generate one or more second images depicting the at least one attribute having the at least one value, at least based on the fact that the inference accuracy of the MLM is less than one or more thresholds for the at least one value, Updating at least one parameter of the at least one MLM using at least one of the second images One or more processing units for performing operations including Equipped with, A system that determines whether to generate one or more additional data samples by analyzing at least one performance metric across attributes.

11. The system according to claim 10, wherein the one or more generating MLMs include a composite generator that generates each of the two second images as a synthesis of the multiple attributes of the one or more attributes.

12. The system according to claim 10, wherein the at least one value of the at least one attribute is one or more values ​​of a first set of attributes, and at least one performance metric corresponds to the relative inference accuracy of the at least one MLM with respect to one or more values ​​of the first set with respect to one or more values ​​of a second set of attributes.

13. The system according to claim 10, wherein at least one performance metric corresponds to the inference accuracy of the at least one MLM of a time-series pattern represented by the values ​​of the at least one attribute across multiple images of the first one or more images, and the at least one value of the at least one attribute includes the time-series pattern.

14. The system according to claim 13, wherein the time-series pattern corresponds to one or more of the frequency, amplitude, velocity, or duration of one or more events represented using the value of the at least one attribute.

15. The aforementioned system Control systems for autonomous or semi-autonomous machines Cognitive systems for autonomous or semi-autonomous machines A system for performing simulation operations. A system for performing digital twin operations. A system for performing deep learning operations. Systems implemented using edge devices, Systems implemented using robots, A system that incorporates one or more virtual machines (VMs). A system that will be implemented, at least partially, in a data center. A system including a collaborative creation platform for three-dimensional (3D) content, or A system that is at least partially implemented using cloud computing resources. The system according to claim 10, which is included in at least one of the following.

Images