Synthetic dataset regeneration for AI systems and applications

Abstract

In various examples, synthetic data set regeneration for AI systems and applications is described herein. For example, the systems and methods described herein may generate a synthetic dataset using a simulator and data (in some examples, referred to as "log data") representing information associated with generating the synthetic dataset by the simulator. For example, the log data may represent at least a parameter used to generate the synthetic data set, a value of the parameter, an asset associated with the parameter, and / or a value representing a result associated with the synthetic data set. The systems and methods may then use the log data to recreate, modify, and / or enhance the synthetic data set. For example, the synthetic data set may be recreated by providing at least log data as input to the simulator such that the simulator regenerates the data set using the same parameters, values, and / or assets.

Description

Background Art

[0001] Many common processing tasks, such as image classification, object segmentation, localization, depth approximation, caption generation, etc., are increasingly implemented using neural networks (or other types of machine learning models). To improve the performance of these neural networks, datasets can be used for training, where the performance of the neural network can be based on the quality of the dataset. The quality of the dataset can be measured based on a comprehensive evaluation of various attributes of the dataset, such as scale, coverage, diversity, accuracy, distribution, and target domain alignment. Of course, a high-quality dataset may require a large amount of data to achieve at least some of these attributes. Accordingly, since simulators can generate large amounts of highly variable data with better ground truth compared to real data at a lower cost, the use of synthetic data has become increasingly common when creating high-quality datasets.

[0002] For example, to generate a dataset for training a neural network, a simulator can receive inputs such as simulation parameters, values of the parameters, assets associated with the parameters, and so on. Then, the simulator can sample the parameters, for example, by randomly selecting values for generating samples (e.g., scenes) associated with the dataset. Thus, since each sample can be generated using random values of the parameters, the samples generated by the simulator for the dataset can be unique. Therefore, although a user can later make a copy of the dataset, such as a copy of a scene representing an object, the user may not be able to regenerate the dataset using the simulator because different values can be sampled. This can cause problems in various situations, such as when the user wants to modify the dataset, enhance the dataset, fix errors in the dataset, etc. Summary of the Invention

[0003] Embodiments of the present disclosure relate to the regeneration of synthetic datasets for artificial intelligence (AI) systems and applications. For example, the systems and methods described herein can use a simulator to generate a dataset and data representing information associated with the dataset generated by the simulator (referred to as "log data" in some examples). For example, the log data can represent at least the parameters used to generate the dataset, the values of the parameters, assets associated with the parameters, and / or values representing results associated with the dataset (e.g., random values derived from the simulation process). Then, the systems and methods can use the log data to recreate, modify, and / or enhance the dataset. For example, the dataset can be recreated by at least providing the log data as an input to the simulator such that the simulator regenerates the dataset using the same parameters, values, and / or assets. If the dataset is to be modified and / or enhanced, at least a portion of the information represented by the log data can be updated and / or data representing new parameters and / or new values can also be input into the simulator.

[0004] Compared to conventional systems such as those described above, in some embodiments, the current system can use log data generated during initial processing performed by the simulator when generating a data set to regenerate the data set. For example, as described above, a conventional system may only be able to generate a copy of the final data set, but a conventional system may not be able to regenerate the data set using the simulator. For similar reasons, compared to conventional systems, in some embodiments, the current system can also enhance the data set (increase randomization, add assets, etc.), modify the data set, generate new samples for the data set (e.g., scenarios), repair the data set (e.g., if the simulator is affected by a bug), and / or provide additional improvements. As will be described in more detail herein, the current system can provide such improvements based on allowing a user to modify the data set used to regenerate the data set and / or allowing the user to provide new parameters and / or values of new parameters when regenerating the data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The present system and method for regenerating synthetic data sets for AI systems and applications will be described in detail below with reference to the accompanying drawings, in which:

[0006] Figure 1A FIG. shows an example data flow diagram of a process for generating log data associated with a simulation, where the log data can later be used to regenerate a data set associated with the simulation;

[0007] Figure 1B FIG. shows an example data flow diagram of a process for regenerating a data set using log data associated with a previous simulation;

[0008] Figure 1C FIG. shows an example data flow diagram of a first process for regenerating a data set using one or more modifications and / or enhancements;

[0009] Figure 1D FIG. shows an example data flow diagram of a second process for regenerating a data set using one or more modifications and / or enhancements;

[0010] Figures 2A - 2B FIG. shows an example of a simulator generating a data set according to some embodiments of the present disclosure;

[0011] Figures 3A - 3B FIG. shows an example of a generated log associated with a simulation according to some embodiments of the present disclosure;

[0012] Figure 4 FIG. shows an example of a data set regenerated to be substantially similar to a previously generated data set according to some embodiments of the present disclosure;

[0013] Figure 5 Shows an example of updating a generation log associated with a simulation according to some embodiments of the present disclosure;

[0014] Figure 6 Shows an example of a modified data set that is regenerated to be similar to a previously generated data set based at least on using modified log data according to some embodiments of the present disclosure;

[0015] Figure 7 Shows an example of a modified data set that is regenerated to be similar to a previously generated data set based at least on using one or more new parameter files according to some embodiments of the present disclosure;

[0016] Figure 8 Is a flowchart showing a method for generating log data according to some embodiments of the present disclosure, which is later used to regenerate a data set associated with a simulation;

[0017] Figure 9 Is a flowchart showing a method for regenerating a data set using log data associated with a previous simulation according to some embodiments of the present disclosure;

[0018] Figure 10A Is an illustration of an example autonomous vehicle according to some embodiments of the present disclosure;

[0019] Figure 10B Is according to some embodiments of the present disclosure Figure 10A An example of the camera position and field of view of an example autonomous vehicle;

[0020] Figure 10C Is according to some embodiments of the present disclosure Figure 10A A block diagram of an example system architecture of an example autonomous vehicle;

[0021] Figure 10D Is for communicating between a cloud-based server and Figure 10A An example system diagram of an example autonomous vehicle;

[0022] Figure 11 Is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure; and

[0023] Figure 12 Is a block diagram of an example data center suitable for implementing some embodiments of the present disclosure. Detailed Description

[0024] Systems and methods related to the regeneration of synthetic datasets for AI systems and applications are disclosed. For example, one or more systems can receive input data that a simulator uses to generate a synthetic dataset. As described herein, the input data can include at least parameter data representing one or more parameter files, asset list data representing one or more asset lists, and / or asset data representing one or more assets. In some examples, a parameter file can include key-value pairs, such as pairs that associate individual parameters with individual values. As described herein, at least a portion of the values can include set values and / or at least a portion of the values can include random values (e.g., distributions, etc.). Additionally, the parameters can include object parameters (e.g., type, count, texture, model, pose, color, etc.), camera parameters (e.g., configuration, lens parameters, resolution parameters, etc.), lighting parameters, scene parameters, output parameters (e.g., dataset name, dataset size, sequence time step, dataset type, etc.), and / or any other type of parameter associated with performing a simulation.

[0025] In some examples, a parameter file can refer to an asset list that stores references to assets (e.g., in text form). For example, an asset list can include a grouping of assets (e.g., objects, camera coordinates, textures, materials, etc.). For example, an asset list associated with a vehicle can include a list of vehicles (also referred to as elements), where, during sampling, a vehicle (e.g., element) can be selected from the list. The selected asset can then be retrieved from the asset data. In some examples, the asset data can be stored on one or more servers accessible to the simulator. In some examples, the assets represented by the asset data can also be associated with a version number. For example, an asset can initially be associated with a first version number (e.g., 1), a first update to the asset can be associated with a second version number (e.g., 2), a second update to the asset can be associated with a third version number (e.g., 3), and so on.

[0026] As described in more detail herein, during a simulation, parameters can be sampled to determine the values associated with the parameters. In some examples, the parameters are sampled in sequence during the simulation. For example, a first parameter can be sampled to determine a first value associated with the first parameter, then a second parameter can be sampled to determine a second value associated with the second parameter, then a third parameter can be sampled to determine a third value associated with the third parameter, and so on. Additionally, asset data associated with the desired assets can be retrieved and used to generate the dataset, where the asset data is retrieved based at least on the sampling of the parameters and / or the values. Then, the simulator can use the parameters, the values of the parameters, and / or the assets to generate the dataset. As described herein, in some examples, the dataset can represent a simulation scene (e.g., an image), a simulation video, simulated text, a simulated system, and / or any other type of simulated data.

[0027] As described herein, to regenerate a dataset, one or more systems can further generate log data representing one or more logs that include information associated with one or more simulations. For example, the logs can represent at least the parameters used by the simulator, the values associated with the parameters, the assets retrieved, the final values generated by the random processes of the simulator (e.g., (e.g., random values derived from the simulation process, such as the value representing the final pose of an object)) and / or any other information. In some examples, the order in which the parameters are written to the logs is the same as the order of the parameters used (e.g., sampled) by the simulator when generating the dataset. For example, using the above example, the first sampled parameter and / or the first value associated with the first parameter can be written, then the second sampled parameter and / or the second value associated with the second parameter can be written, then the third sampled parameter and / or the third value associated with the third parameter can be written, and so on.

[0028] Then, one or more systems can regenerate the dataset (and / or a portion of the dataset, such as a particular scenario) by at least using the log data to perform one or more simulations again using the simulator. For example, one or more systems can input the log data (and / or the portion of the log data associated with that portion of the dataset) into the simulator. Then, the simulator can process the log data to regenerate the dataset using the same parameters, values, and / or assets as were used in the original simulation to generate the original dataset. In some examples, since the parameters can be written to the log in the same order as the parameters used by the simulator during the original simulation, the simulator can sample the parameters again in the same order during the new simulation, which can improve the accuracy of the regenerated dataset.

[0029] In some examples, one or more systems can further regenerate a dataset through modification and / or enhancement. As described herein, modification can include changing parameters associated with the dataset, such as camera configuration, object pose (e.g., position, orientation, etc.), object type, etc. Additionally, enhancement can include adding new assets, such as new objects, textures, materials, and so on. In some examples, the system can regenerate a dataset with modifications and / or enhancements based at least on user and / or device updates to one or more parameters, one or more values, and / or one or more assets represented by the log data. In such examples, the system can then input the updated log data into the simulator to regenerate the modified and / or enhanced dataset. In some examples, the system can regenerate a dataset with modifications and / or enhancements based at least on generating new parameter data (e.g., one or more new parameter files) representing one or more new parameters and / or one or more values associated with the new parameters. In such examples, the system can then input the log data and the new parameter data into the simulator to regenerate the modified and / or enhanced dataset.

[0030] In some examples, when regenerating a dataset (and / or a modified and / or enhanced dataset), the system can generate log data again representing one or more logs that include information associated with one or more simulations. For example, the log can represent at least the parameters used by the simulator, the values associated with the parameters, the assets retrieved, the final values produced by the random processes of the simulator (e.g., values representing the final pose of an object), and / or any other information associated with the regeneration. Thus, the system can continue to use the log data to regenerate, modify, and / or enhance the dataset again.

[0031] As described herein, performing these processes can provide several improvements over conventional systems. For a first example, the processes described herein can allow for enhancing and / or modifying existing synthetic datasets, such as by adding new assets (e.g., objects) to an existing scene, changing the texture of an object, modifying the camera configuration, modifying the lighting, etc. Additionally, for similar reasons, new annotations and / or ground truths can be added to existing synthetic datasets, which can allow for training and / or evaluating different tasks. For example, these processes can allow for adding a more refined semantic layer to a segmentation mask of an object and / or allowing a user to add new features that can produce new annotations associated with the synthetic dataset.

[0032] The processes described herein can also allow for regenerating synthetic datasets that are streamed directly to a network without being written to disk (e.g., the synthetic dataset cannot be copied). For example, regenerating a synthetic dataset that is streamed directly to a network can allow for rechecking training content using the regenerated synthetic dataset as needed after generation.

[0033] The processes described herein may also allow for the regeneration of synthetic data sets that are affected by vulnerabilities (e.g., in a simulator) that cause problems with the synthetic data sets. For example, regenerating a synthetic data set that is affected by an unexpected phenomenon (e.g., an anomaly, a vulnerability, an artifact, or other problem) may allow for the fixing of desired attributes associated with the affected synthetic data set.

[0034] The processes described herein may also allow for a verification step to test the integrity of a software build and a hardware configuration. For example, a regenerated synthetic data set may be compared to a previously generated synthetic data set to check for problems that may have occurred with the software build and / or the hardware configuration. For example, if a regenerated synthetic data set does not substantially match a previously generated synthetic data set (e.g., one or more objects are depicted differently, such as by using different colors), this may indicate a problem with the software build and / or the hardware configuration.

[0035] As described herein, different files may be used for different aspects of the present invention, such as parameter files, asset lists, asset files, generation logs, and the like. In some examples, any type of file may be used, such as but not limited to YAML files, Universal Scene Description (USD) files, Portable Network Graphics (PNG) files, Printer Font Metrics (PFM) files, image files, text files, Graphics Interchange Format (GIF) files, Portable Network Graphics (PNG) files, Portable Document Format (PDF) files, and / or any other type of file format.

[0036] The systems and methods described herein may be used by, but are not limited to, the following: non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more Adaptive Driver Assistance Systems (ADAS)), manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, airships, boats, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, airplanes, engineering vehicles, submarines, drones, and / or other vehicle types. Additionally, the systems and methods described herein may be used for a variety of purposes, such as but not limited to machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and surveillance, simulation and digital twins, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation, and / or digital twins, data center processing, conversational AI, light transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation of 3D assets, cloud computing, and / or any other suitable application.

[0037] The disclosed embodiments can be included in various different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines), systems implemented using robots, aviation systems, medical systems, boating systems, intelligent area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems implementing large language models (LLMs), systems containing one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in a data center, systems for performing conversational AI operations, systems for performing optical transmission simulations, systems for performing collaborative content creation of 3D assets, systems implemented at least partially using cloud computing resources, and / or other types of systems.

[0038] Reference Figure 1A , Figure 1A FIG. shows an example data flow diagram of a process 100 for generating log data associated with a simulation according to some embodiments of the present disclosure, where the log data can be later used to regenerate a dataset associated with the simulation. It should be understood that such arrangements and other arrangements described herein are presented only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, function groupings, etc.) can be used in addition to or in place of the shown arrangements and elements, and some elements can be entirely omitted. Moreover, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in combination with other components, and can be implemented in any suitable combination and location. The various functions performed by the entities described herein can be implemented by hardware, firmware, and / or software. For example, the various functions can be implemented by a processor executing instructions stored in a memory. In some embodiments, the systems, methods, and processes described herein can use components, features, and / or functions similar to those of the example autonomous vehicle 1000 in Figures 10A - 10D , the example computing device 1100 in Figure 11 , and / or Figure 12 the example data center 1200 in

[0039] Process 100 may include a parsing component 102 receiving one or more parameter files 104 and / or one or more asset lists 106. As described herein, parameter file 104 may include key-value pairs, such as pairs associating individual parameters with individual values. In some examples, the values may include set values (e.g., raw values) associated with deterministic parameters or random values (e.g., distributions) associated with random parameters. Additionally, set values may include numbers, strings, tuples, etc., while random values may include normals, ranges, selections, walks, etc. In some examples, parameters may include object parameters (e.g., type, count, texture, model, pose, color, etc.), camera parameters (e.g., configuration, lens parameters, resolution parameters, etc.), lighting parameters, scene parameters, output parameters (e.g., dataset name, dataset size, sequence time step, dataset type, etc.), and / or any other type of parameter associated with the simulation. In some examples, parameters may be grouped into parameter groups. For example, a first group may include chaotic random objects, a second group may include realistic objects, a third group may include one type of object (e.g., people, vehicles, etc.), and so on.

[0040] Asset list 106 may include a list of assets, such as in the form of a text file, where one or more parameter files 104 reference asset list 106. As described herein, assets may include, but are not limited to, objects, textures, materials, camera coordinates, and / or any other type of asset. For example, asset list 106 includes a list of objects that may be included in the simulation, where asset list 106 includes text paths to files associated with the object assets on one or more asset servers 108. As another example, asset list 106 may include a list of a particular type of object (e.g., people) that may be included in the simulation, where asset list 106 again includes text paths to files associated with the people assets on one or more asset servers 108. As shown, process 100 may include parsing component 102 at least combining one or more parameter files 104 with one or more asset lists 106 and then sending the parameter files and asset list 106 to simulation component 110 as input.

[0041] Process 100 may include one or more asset servers 108 inputting asset data 112 as input into simulation component 110. For example, asset server 108 may store files (e.g., USD files) representing assets available for simulation. As described herein, an asset may include physically accurate two-dimensional (2D) and / or three-dimensional (3D) objects that contain accurate physical properties, behaviors, and connected data streams to represent the real world in a simulated digital world. In some examples, an asset may be associated with a version number. For example, an asset may initially be associated with an initial version number, such as version one. After the asset is updated, the updated asset may be associated with a second version number, such as version two. As the asset continues to be updated, this may continue to repeat, where these version numbers are important for the regeneration of the data set, as described in more detail herein. Thus, asset data 112 may represent one or more assets that simulation component 110 uses as input.

[0042] As described herein, simulation component 110 may include one or more simulators configured to receive input, perform one or more simulations (e.g., perform generation) using the input, and then output one or more data sets 114 associated with the one or more simulations. In some examples, one or more data sets 114 may include simulation scenarios (e.g., images), simulation videos, simulation text, simulation systems, and / or any other type of simulation data. For a first example, if simulation component 110 is configured to generate a simulation associated with an object falling, data set 114 may include one or more scenarios depicting the object from at least the time it starts to fall until the time it finally comes to rest. For a second example, if simulation component 110 is configured to generate a simulation associated with a vehicle traveling in an environment, data set 114 may include one or more scenarios depicting objects (e.g., vehicles, pedestrians, traffic signals, etc.) located in the environment where the vehicle travels over a period of time.

[0043] In some examples, simulation component 110 may be configured to generate multiple simulations. As described herein, the number of simulations may include but is not limited to one simulation, five simulations, ten simulations, fifty simulations, one hundred simulations, one thousand simulations, ten thousand simulations, and / or any other number of simulations. For example, and again, if simulation component 110 is configured to generate a simulation associated with an object falling, simulation component 110 may perform multiple simulations using the input to generate different scenarios depicting the object falling until it comes to rest at various positions within the environment. In such an example, the positions may differ from each other based at least on each simulation including different parameters and / or different parameter values.

[0044] For example, Figure 2AShows a first example of a simulation component 110 performing a simulation to generate a data set 202 (which may represent and / or include data set 114) in accordance with some embodiments of the present disclosure. As shown, the data set 202 may include a plurality of scenarios 204(1)-(3) (also singularly referred to as "scenario 204" or plurally as "plural scenarios 204"), where each scenario 204 is associated with a specific time instance. For example, the first scenario 204(1) may be associated with a first time instance in which a first object 206(1) (e.g., a ball) drops from a position within the environment 208. Additionally, the second scenario 204(2) may be associated with a second time instance in which the first object 206(1) contacts a second object 206(2) also located within the environment 208. Finally, the third scenario 204(3) may be associated with a third time instance in which the first object 206(1) reaches a stationary position within the environment 208.

[0045] In addition, Figure 2B Shows a second example of a simulation component 110 performing a simulation to generate a data set 210 (which may represent and / or include data set 114) in accordance with some embodiments of the present disclosure. As shown, the data set 210 may again include a plurality of scenarios 212(1)-(3) (also singularly referred to as "scenario 212" or plurally as "plural scenarios 212"), where each scenario 212 is associated with a different simulation result. For example, the first scenario 212(1) may be associated with a vehicle 214 navigating in an environment 216, where another vehicle 218 is traveling towards the vehicle 214 in the oncoming lane. Additionally, the second scenario 212(2) may be associated with the vehicle 214 again navigating in the environment 216, but where another vehicle 220 drives in front of the vehicle 214. Additionally, the third scenario 212(3) may be associated with the vehicle 214 again navigating in the environment 216, but where a pedestrian 222 is crossing the road. In some examples, the data set 210 may be used to train one or more systems associated with the vehicle (e.g., a perception system, a positioning system, a navigation system, etc.).

[0046] Return to reference Figure 1AIn an example, process 100 may include simulation component 110 using sampling component 116 to at least process parameter file 104 to sample values associated with parameters of the simulation. For example, for a simulation, sampling component 116 may use one or more parameter files 104 to determine a first parameter associated with the simulation. Then, sampling component 116 may use sampling to determine a first value associated with the first parameter. As described herein, the first value may include a set value, and / or the first value may be selected from a distribution associated with the first parameter. Then, sampling component 116 may use one or more parameter files 104 to determine a second parameter associated with the simulation. Then, sampling component 116 may use sampling to determine a second value associated with the second parameter. As described herein, the second value may include a set value, and / or the second value may be selected from a distribution associated with the second parameter. Then, sampling component 116 may continue to perform these processes for one or more additional parameters represented by one or more parameter files 104.

[0047] In some examples, the selected values may be associated with assets (e.g., elements) from one or more asset lists 106. Additionally, the assets may be associated with an object model stored by one or more asset servers 108. In this way, simulation component 110 may retrieve asset data 112 (e.g., files) associated with the object model from one or more asset servers 108. In this way, simulation component 110 may use the object model when generating a simulation.

[0048] Then, process 100 may include simulation component 110 using generation component 118 to perform the simulation (e.g., perform generation associated with the simulation). For example, to perform the simulation, generation component 118 may use the parameters and / or values of the parameters determined by sampling component 116, where the parameters and / or values may be represented by 120. Additionally, to perform the simulation, generation component 118 may use the assets represented by asset data 112. Then, generation component 118 may use any technique to generate a simulation using the parameters, values, and assets.

[0049] Then, process 100 may include capture component 122 capturing data, such as a simulation scene (e.g., an image), simulation video, simulation text, simulation system, and / or any other type of simulation data associated with the simulation generated by generation component 118. For a first example, if generation component 118 generates a simulation associated with an object falling, capture component 122 may capture at least from the time the object starts to fall to the time the object finally comes to rest (e.g., from Figure 2AExample scenario 204) depicts one or more scenarios of an object. For a second example, if generation component 118 generates a simulation associated with vehicle travel, capture component 122 can capture one or more scenarios (e.g., Figure 2B scenario 212) in the example of. In any example, capture component 122 can output one or more data sets 114 representing data captured from the simulation.

[0050] Process 100 can include simulation component 110 (and / or another system and / or component) generating one or more generation logs 124 associated with the simulation. As described herein, generation log 124 can later be used to re - execute the simulation in order to regenerate one or more data sets 114. For example, generation log 124 can at least represent parameters used by simulation component 110, values associated with the parameters, retrieved assets, final values obtained from the random processes of simulation component 110 (e.g., random values derived from the simulation process, such as values representing the final pose of an object) and / or any other information associated with the simulation. In some examples, at least some of the parameters are written to generation log 124 in the same order as the parameters used (e.g., sampled) by simulation component 110 when generating data set 114. For example, the first sampled parameter and / or the first value associated with the first parameter can be written, then the second sampled parameter and / or the second value associated with the second parameter can be written, then the third sampled parameter and / or the third value associated with the third parameter can be written, and so on.

[0051] For example, Figure 3A illustrates, according to some embodiments of the present disclosure, a simulation (e.g., generation Figure 2AThe first example of the generation log 302 associated with the simulation of the third scenario 204(3) in the example of (which may represent and / or include the generation log 124). As shown, the generation log 302 may associate the values 304(1)-(4) with the parameters 306(1)-(4) used to generate the third scenario 204(3). For example, the first parameter 306(1) may be associated with the type of the first object 206(1), the second parameter 306(2) may be associated with the type of the second object 206(2), the third parameter 306(3) may be associated with the lighting, and the fourth parameter 306(4) may be associated with the camera configuration. Thus, at least the first value 304(1) may be associated with a ball, and the second value 304(2) may be associated with a pyramid. As described herein, in some examples, the order associated with the values 304(1)-(4) and / or the parameters 306(1)-(4) may be based on the order in which the parameters 306(1)-(4) are sampled during the simulation. For example, to generate the third scenario 204(3), the first parameter 306(1) may have been sampled, followed by the second parameter 306(2), followed by the third parameter 306(3), and finally the fourth parameter 306(4).

[0052] In addition, the generation log 302 may also include the assets 308(1)-(2) used to generate the third scenario 204(3) and the version numbers 310(1)-(2) associated with the assets 308(1)-(2). For example, the first asset 308(1) may be associated with the first object 206(1), and the version number 310(1) may indicate the version of the first object 206(1). In addition, the second asset 308(2) may be associated with the second object 206(2), and the version number 310(2) may indicate the version of the second object 206(2). In addition, the generation log 302 may include one or more values 312 associated with the final pose 314 of at least the first object 206(1). As described herein, one or more values may include random values generated by the simulation component 102 when performing the simulation. For example, the value 312 may indicate the position (e.g., x coordinate position, y coordinate position, and z coordinate position) and pose (e.g., roll, pitch, and yaw) associated with the first object 206(1) in the third scenario 204(3), where the position and / or pose are random because the first object 206(1) falls.

[0053] Figure 3B Shows a simulation according to some embodiments of the present disclosure (e.g., generation Figure 2BSecond example of the generation log 316 associated with the simulation of the third scenario 212(3) in the example of

[0054] In addition, the generation log 316 may also include assets 322(1)-(2) for generating the third scenario 212(3) and version numbers 324(1)-(2) associated with the assets 322(1)-(2). For example, the first asset 322(1) may be associated with the vehicle 214, and the version number 324(1) may indicate the version of the vehicle 214. In addition, the second asset 322(1) may be associated with the pedestrian 222, and the version number 324(2) may indicate the version of the pedestrian 222.

[0055] Although Figures 3A - 3B the examples of

[0056] Return reference Figure 1A It should be noted that the text seems to be incomplete or have some formatting issues in its original form, which might affect the full comprehensibility of the translation in the context. But the translation is done according to the requirements.As an example, process 100 may include associating at least a portion of generated log 124 with one or more data sets 114. In some examples, process 100 may include associating each of generated logs 124 with one or more data sets 114. In some examples, process 100 may include associating portions of generated log 124 with portions of data set 114. For example, if generated log 124 is generated for a scenario of data set 114, generated log 124 may be associated with that scenario. As described herein, generated log 124 may then be used to regenerate data set 114.

[0057] For example, Figure 1B FIG. shows an example data flow diagram of process 126 for regenerating one or more data sets 114 using one or more generated logs 124 associated with one or more simulations, according to some embodiments of the present disclosure. As described herein, in some examples, generated log 124 may be used to regenerate one or more data sets 128 that are substantially similar to one or more original data sets 114 generated during one or more initial simulations. In such examples, to regenerate data set 114, simulation component 110 may receive generated log 124 and asset data 112 representing the assets used to generate data set 114 as inputs. In some examples, if regenerating is associated with a portion of data set 114, such as a scenario, simulation component 110 may receive generated log 124 associated with the scenario and asset data 112 representing the assets used to generate the scenario. Additionally, when regenerating data set 114, simulation component 110 may not receive one or more parameter files 104 and / or one or more asset lists 106.

[0058] For example, to regenerate data set 112, generation component 118 may execute the simulation again using the parameters and / or values of the parameters indicated by generated log 124. As described herein, to perform a regeneration similar to the original generation, generation component 118 may read the parameters and / or values of the parameters in the order indicated in generated log 124, where the order matches the original order in the original simulation. For example, referring to Figure 3AIn an example, the generation component 118 can read a first value 304(1) of a first parameter 306(1), then read a second value 304(2) of a second parameter 306(2), then read a third value 304(3) of a third parameter 306(3), and finally read a fourth value 304(4) of a fourth parameter 306(4). Additionally, the generation component 118 can retrieve asset data 112 representing an asset. For example, the simulation component 110 can retrieve a version number 310(1) of a first asset 308(1) and a version number 310(2) of a second asset 308(2). Further, in some examples, the generation component 118 can use one or more final poses of one or more objects. For example, the generation component 118 can use a value 312 of a pose 314 to determine a final pose of a first asset 308(1).

[0059] Then, the process 126 can include the capture component 122 capturing data, such as a simulation scenario (e.g., an image), simulation video, simulation text, a simulation system, and / or any other type of simulation data associated with the simulation generated by the simulation component 110. Then, the capture component 122 can output a data set 128 representing the captured data. As described herein, in an example where the simulation component 110 is configured to regenerate the data set 114 using the original generation log 124, the data set 128 can substantially match the data set 114.

[0060] For example, Figure 4 illustrates an example of a data set according to some embodiments of the present disclosure that is regenerated to be substantially similar to a previously generated data set. As shown, by performing Figure 1B the process 126, the simulation component 110 can generate a new data set that at least includes a new scene 402 that is substantially similar to a third scene 204(3). For example, both the scene 402 and the third scene 204(3) include the same objects 206(1)-(2) located within an environment 208. Additionally, both the scene 402 and the third scene 204(3) include a first object 206(1) in a substantially same end pose (e.g., using a random value). In some examples, the simulation component 110 can perform a similar process to also regenerate at least one of the scenes 204(1)-(2) and / or at least one of the scenes 212(1)-(3).

[0061] Returning to the reference Figure 1BIn an example, process 126 may include simulation component 110 (and / or another system and / or component) generating one or more generation logs 130 associated with the simulation. As described herein, generation log 130 may later be used to re - execute the simulation to regenerate dataset 114 and / or dataset 128. For example, generation log 130 may at least represent parameters used by simulation component 110, values associated with the parameters, retrieved assets, final values obtained by a random process of simulation component 110 (e.g., a value representing the final pose of an object), and / or any other information. In some examples, at least some parameters are written to generation log 130 in the same order as the parameters used (e.g., sampled) by simulation component 110 when generating dataset 128. For example, the first sampled parameter and / or the first value associated with the first parameter may be written, then the second sampled parameter and / or the second value associated with the second parameter may be written, then the third sampled parameter and / or the third value associated with the third parameter may be written, and so on. Additionally, since process 126 is associated with generating dataset 128 which is substantially similar to dataset 112, generation log 130 may be substantially similar to generation log 124.

[0062] Figure 1C FIG. shows an example data - flow diagram of a first process 132 for regenerating dataset 114 using one or more modifications and / or enhancements according to some embodiments of the present disclosure. For example, as described herein, in some examples, to modify and / or enhance an original simulation, one or more users and / or systems may update generation log 124. For example, as shown, update component 134 may update one or more parameters in generation log 124, update one or more values in generation log 124, remove one or more parameters from generation log 124, add one or more parameters to generation log 124, update one or more assets in generation log 124, update one or more version numbers in the generation log, remove one or more assets from generation log 124, and / or add one or more assets to generation log 124 to generate one or more updated generation logs 136. In some examples, update component 134 performs at least a portion of the update based at least on receiving one or more inputs from one or more users. In some examples, update component 134 automatically performs at least a portion of the update, e.g., by using one or more scripts, applications, etc., executed by update component 134.

[0063] For example, Figure 5 FIG. shows an example of updating a generation log 302 associated with a simulation to generate an updated generation log 502 according to some embodiments of the present disclosure. As Figure 5As shown in the example of, the update component 134 can update at least the second value 304(2) associated with the second parameter 306(2) to include the value 504 and update the version number 310(1) associated with the first asset 308(1) to include the new version number 506. However, in other examples, the update component 134 can also remove one or more parameters, add one or more parameters, update the value 312 associated with the final pose 314, and / or perform any other updates.

[0064] Return reference Figure 1C In the example of, to regenerate the dataset 112 with modifications and / or enhancements, the generation component 118 can use the parameters and / or the values of the parameters indicated by the generation log 136 to perform the simulation again. As described herein, to perform a regeneration similar to the original generation, the generation component 118 can read the parameters and / or the values of the parameters in the order in the generation log 136, where the order at least partially matches the original order in the original simulation. For example, referring to Figure 5 In the example of, the generation component 118 can read the first value 304(1) of the first parameter 306(1), then read the value 504 of the second parameter 306(2), then read the third value 304(3) of the third parameter 306(3), and finally read the fourth value 304(4) of the fourth parameter 306(4).

[0065] In addition, the generation component 118 can retrieve the asset data 138 representing one or more assets, where the assets can be changed and / or modified at least based on the generation log 136. For example, the simulation component 110 can retrieve the new version 506 of the first asset 308(1) and the version number 310(2) of the second asset 308(2). In addition, in some examples, the generation component 118 can use the one or more final poses of one or more objects. For example, the generation component 118 can use one or more values 312 of the pose 314 to determine the final pose of the first asset 308(1).

[0066] Then, the process 132 can include the capture component 122 capturing data, such as a simulation scene (e.g., an image), a simulation video, simulation text, a simulation system, and / or any other type of simulation data associated with the simulation generated by the simulation component 110. Then, the capture component 112 can output one or more datasets 140 representing the captured data. As described herein, in an example where the simulation component 110 is configured to regenerate the dataset 114 with modifications and / or enhancements, the dataset 140 can still be similar to the dataset 114, but includes modifications and / or enhancements.

[0067] For example, Figure 6Shows an example of a modified data set regenerated to be similar to a previously generated data set according to some embodiments of the present disclosure. As shown, by performing Figure 1C process 132, simulation component 110 can generate a new data set that at least includes a new scenario 602 related to the third scenario 204(3) but with modifications. For example, scenario 602 includes a new object 604 that is a modified version of the second object 206(2). In some examples, the modification can be based on a new value 504 of the second parameter 306(2). Additionally, scenario 602 includes a new object 606 that is an updated version of the first object 206(1). In some examples, the modification can be based on a new version 506 of the first asset 308(1). Also, both scenario 602 and the third scenario 204(3) include the object 606 and the first object 206(1) because they include substantially similar poses. In some examples, simulation component 110 can perform a similar process to also regenerate at least one of scenarios 204(1)-(2) and / or at least one of scenarios 212(1)-(3) with one or more modifications and / or one or more enhancements.

[0068] Returning to the example Figure 1C process 132 can include simulation component 110 (and / or another system and / or component) generating one or more generation logs 142 associated with the simulation. As described herein, the generation log 142 can later be used to re - execute the simulation to regenerate the data set 140. For example, the generation log 142 can at least represent the parameters used by the simulation component 110, the values associated with the parameters, the retrieved assets, the final values obtained by the random processes of the simulation component 110 (e.g., values representing the final poses of objects), and / or any other information. In some examples, at least some of the parameters are written to the generation log 142 in the same order as the parameters used (e.g., sampled) by the simulation component 110 when generating the data set 140. For example, the first sampled parameter and / or the first value associated with the first parameter can be written, then the second sampled parameter and / or the second value associated with the second parameter can be written, then the third sampled parameter and / or the third value associated with the third parameter can be written, and so on.

[0069] Figure 1D Shows an example data flow diagram of a second process 144 for regenerating a data set 114 using one or more modifications and / or enhancements. As Figure 1DAs shown in the example of , process 144 may include parsing component 102 receiving one or more new parameter files 146 and / or one or more new asset lists 148 associated with the regeneration of dataset 114. For example, parameter file 146 may include one or more new parameters, one or more new values of the new parameters, one or more modified values of the parameters associated with generation log 124, and / or the like associated with regeneration. Then, process 144 may include parsing component 102 sending parameter file 146 and / or asset list 148 to simulation component 110.

[0070] Accordingly, the inputs associated with regeneration may include generation log 124, parameter file 146, asset list 148, and / or asset data 138 representing one or more assets associated with the simulation. For example, generation component 118 may use the parameters and / or values of the parameters in generation log 124 and / or the parameters and / or values of the parameters in parameter file 146 (which may be represented by 152) to perform the simulation again, but with modifications and / or enhancements. As described herein, to perform a regeneration similar to the original generation, generation component 118 may read the parameters and / or values of the parameters in the order in generation log 124 and / or parameter file 146, where the order at least partially matches the original order of the original simulation. Generation component 118 may also retrieve asset data 150 representing the assets required to perform the simulation.

[0071] Then, process 144 may include capture component 122 capturing data, such as a simulation scenario (e.g., an image), simulation video, simulation text, simulation system, and / or any other type of simulation data associated with the simulation generated by simulation component 110. Then, capture component 112 may output one or more datasets 154 representing the captured data. As described herein, in an example where simulation component 110 is configured to regenerate dataset 114 with modifications and / or enhancements, dataset 154 may still be similar to dataset 114, but includes modifications and / or enhancements.

[0072] For example, Figure 7 illustrates an example of regenerating a modified dataset similar to a previously generated dataset using one or more new parameter files. As shown, by performing Figure 1DIn process 144, the simulation component 110 may generate a new data set that includes at least a new scene 702 that is similar to the third scene 204(3) but with enhancements. For example, both scene 702 and the third scene 204(3) include the same objects 206(1)-(2) located within the environment 208. Additionally, both scene 702 and the third scene 204(3) include a first object 206(1) in substantially the same end pose. Further, scene 702 may include a new object 704 in a new pose within the environment 208, where the new object 704 is based on a new parameter file. In some examples, the simulation component 110 may perform a similar process to also regenerate at least one of scenes 204(1)-(2) and / or at least one of scenes 212(1)-(3).

[0073] Return reference Figure 1D In an example, process 144 may include the simulation component 110 (and / or another system and / or component) generating one or more generation logs 156 associated with the simulation. As described herein, the generation logs 156 may later be used to re-execute the simulation to regenerate the data set 154. For example, the generation logs 156 may at least represent the parameters used by the simulation component 110, the values associated with the parameters, the assets retrieved, the final values obtained by the random processes of the simulation component 110 (e.g., values representing the final poses of objects), and / or any other information. In some examples, at least some of the parameters are written to the generation logs 156 in the same order as the parameters used (e.g., sampled) by the simulation component 110 when generating the data set 154. For example, the first sampled parameter and / or the first value associated with the first parameter may be written, then the second sampled parameter and / or the second value associated with the second parameter may be written, then the third sampled parameter and / or the third value associated with the third parameter may be written, and so on.

[0074] Now refer to Figure 8 and Figure 9 Each block of methods 800 and 900 described herein includes a computational process that may be performed using any combination of hardware, firmware, and / or software. For example, various functions may be implemented by a processor executing instructions stored in a memory. Methods 800 and 900 may also be embodied as computer-usable instructions stored on a computer storage medium. Methods 800 and 900 may be provided by a stand-alone application, a service, or a hosted service (independently or in combination with another hosted service) or a plug-in of another product, to name a few. Additionally, methods 800 and 900 are described by way of example with respect to Figures 1A - 1D However, these methods 800 and 900 may additionally or alternatively be performed by any one system or any combination of systems, including but not limited to the systems described herein.

[0075] Figure 8It is a flowchart showing a method 800 for generating log data to regenerate a dataset associated with a simulation according to some embodiments of the present disclosure. At block B802, method 800 may include: determining one or more parameters associated with generating at least a portion of a dataset. For example, simulation component 110 may determine one or more parameters associated with generating at least a portion of dataset 114. As described herein, in some examples, simulation component 110 may determine the one or more parameters based at least on parameter file 104. For example, parameter file 104 may represent parameters for performing a simulation. Additionally, as described herein, parameters may include object parameters (e.g., type, count, texture, model, pose, color, etc.), camera parameters (e.g., configuration, lens parameters, resolution parameters, etc.), lighting parameters, scene parameters, output parameters (e.g., dataset name, dataset size, sequence time steps, dataset type, etc.), and / or any other type of parameter associated with the simulation.

[0076] At block B804, method 800 may include: determining one or more values associated with the one or more parameters. For example, simulation component 110 may determine one or more values associated with the one or more parameters. As described herein, in some examples, simulation component 110 may determine the one or more values based at least on parameter file 104. For example, parameter file 104 may include key-value pairs, such as pairs associating parameters with values. In some examples, simulation component 110 determines the one or more values by sampling the one or more parameters in sequence. For example, simulation component 110 may sample a first parameter to determine a first value for the first parameter, then sample a second parameter to determine a second value for the second parameter, then sample a third parameter to determine a third value for the third parameter, and so on.

[0077] At block B806, method 800 may include: generating at least a portion of the dataset based at least on the one or more values. For example, simulation component 110 may generate at least a portion of dataset 114 based at least on the one or more values of the one or more parameters. As described herein, at least a portion of dataset 114 may include one or more simulation scenarios (e.g., images), one or more simulation videos, simulated text, one or more simulated systems, and / or any other type of simulation data associated with the simulation. Additionally, in some examples, simulation component 110 may use asset data 112 representing one or more assets associated with the simulation to generate at least a portion of dataset 114.

[0078] At block B808, method 800 may include: generating data representing at least one or more values associated with one or more parameters and corresponding at least to a state or time associated with at least a portion of an associated simulated data set. For example, simulation component 110 may generate data representing generation log 124, where generation log 124 includes at least one or more values of one or more parameters. In some examples, generation log 124 may include one or more values of one or more parameters in an order that is the same as the order in which the parameters are sampled during the simulation. Additionally, as described herein, in some examples, generation log 124 may also indicate an asset used to generate the at least a portion of data set 114, a version associated with the asset, one or more values associated with one or more poses of one or more objects, and / or one or more random values associated with the simulation (e.g., one or more values indicating a final pose of an object).

[0079] Additionally, in some examples, generation log 124 may correspond to a state and / or time associated with the at least a portion of the data set. For example, generation log 124 may be associated with one or more instances (e.g., scenes) of the data set, where the one or more instances are associated with a time and / or state of the simulation. For example, one or more scenes (e.g., each scene) of the data set may correspond to a respective time and / or state. Thus, as described herein, generation log 124 may be associated with such a time and / or state.

[0080] Figure 9 is a flowchart showing a method 900 of regenerating a data set using log data associated with a simulation according to some embodiments of the present disclosure. At block B902, method 900 may include: determining at least a portion of a first data set 114 for recreation. For example, simulation component 110 may determine the at least a portion of the first data set for recreation. As described herein, the at least a portion of first data set 114 may include one or more simulation scenes (e.g., images), one or more simulation videos, simulated text, one or more simulation systems, and / or any other type of simulation data associated with the simulation. In some examples, simulation component 110 determines the at least a portion of first data set 114 based at least on user input.

[0081] At block B904, method 900 may include: obtaining data representing one or more values associated with one or more parameters used to generate the at least a portion of the first data set. For example, simulation component 110 may receive data representing one or more values associated with one or more parameters used to generate the at least a portion of the first data set 114. As described herein, the data may represent a generation log 124 associated with the first data set 114 and / or an updated generation log 142 modified by one or more users and / or systems. In some examples, simulation component 110 may also receive a new parameter file 146 and / or a new asset list 148 associated with the regeneration, such as when the regeneration includes modifications and / or enhancements.

[0082] At block B906, method 900 may include: generating at least a portion of a second data set related to the at least a portion of the first data set based at least on the one or more values represented by the data. For example, simulation component 110 may perform the regeneration using at least the data representing the generation log 124 and / or the generation log 142. In some examples, simulation component 110 may perform the regeneration by at least sampling one or more values of one or more parameters based at least on an order associated with the generation log 124 and / or the generation log 142. Additionally, in some examples, simulation component 110 may use the asset data 112, the parameter file 146, the asset list 148, and / or the asset data 150 to perform the regeneration.

[0083] Example autonomous vehicle

[0084] Figure 10AFIG. 0 is an illustration of an example autonomous vehicle 1000 in accordance with some embodiments of the present disclosure. The autonomous vehicle 1000 (alternatively, referred to herein as "vehicle 1000") can include, but is not limited to, passenger vehicles such as cars, trucks, buses, first responder vehicles, shuttle vehicles, electric or motorized bicycles, motorcycles, fire trucks, police vehicles, ambulances, boats, construction vehicles, underwater vessels, robotic vehicles, drones, airplanes, vehicles connected to trailers (e.g., semi-trailer trucks for hauling goods) and / or another type of vehicle (e.g., driverless and / or accommodating one or more passengers). Autonomous vehicles are generally described according to the levels of automation defined by the National Highway Traffic Safety Administration (NHTSA), a division of the U.S. Department of Transportation, and the Society of Automotive Engineers (SAE) in "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles" (Standard No. J3016-201806, issued June 15, 2018, Standard No. J3016-201609, issued September 30, 2016, and previous and future versions of the standard). The vehicle 1000 may be capable of implementing one or more functions corresponding to levels 3-5 of the autonomous driving level. The vehicle 1000 may be capable of implementing one or more functions corresponding to levels 1-5 of the autonomous driving level. For example, depending on the embodiment, the vehicle 1000 may be capable of providing driver assistance (level 1), partial automation (level 2), conditional automation (level 3), high automation (level 4), and / or full automation (level 5). The term "autonomous" as used herein may include any and / or all types of autonomy of the vehicle 1000 or other machines, such as fully autonomous, highly autonomous, conditionally autonomous, partially autonomous, providing assisted autonomy, semi-autonomous, primarily autonomous, or other designations.

[0085] The vehicle 1000 can include components such as a chassis, a body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components of the vehicle. The vehicle 1000 can include a propulsion system 1050, such as an internal combustion engine, a hybrid power plant, an all-electric motor, and / or another type of propulsion system. The propulsion system 1050 can be connected to a driveline of the vehicle 1000 that can include a transmission to effect propulsion of the vehicle 1000. The propulsion system 1050 can be controlled in response to receiving a signal from the throttle / accelerator 1052.

[0086] A steering system 1054 that may include a steering wheel can be used to steer the vehicle 1000 (e.g., along a desired path or route) while the propulsion system 1050 is operating (e.g., while the vehicle is in motion). The steering system 1054 may receive signals from a steering actuator 1056. For fully autonomous (Level 5) functionality, the steering wheel may be optional.

[0087] A brake sensor system 1046 can be used to operate vehicle brakes in response to receiving signals from a brake actuator 1048 and / or a brake sensor.

[0088] One or more controllers 1036 that may include one or more system-on-chips (SoCs) 1004 ( Figure 10C ) and / or one or more GPUs can provide signals (e.g., representing commands) to one or more components and / or systems of the vehicle 1000. For example, one or more controllers can send signals to operate the vehicle brakes via one or more brake actuators 1048, operate the steering system 1054 via one or more steering actuators 1056, and operate the propulsion system 1050 via one or more throttles / accelerators 1052. One or more controllers 1036 can include one or more on-board (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operation commands (e.g., signals representing commands) to enable autonomous driving and / or assist a human driver in driving the vehicle 1000. One or more controllers 1036 can include a first controller 1036 for autonomous driving functionality, a second controller 1036 for functional safety functionality, a third controller 1036 for artificial intelligence functionality (e.g., computer vision), a fourth controller 1036 for infotainment functionality, a fifth controller 1036 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 1036 can handle two or more of the above functions, two or more controllers 1036 can handle a single function, and / or any combination thereof.

[0089] One or more controllers 1036 may provide indications for controlling one or more components and / or systems of vehicle 1000 in response to sensor data (e.g., sensor inputs) received from one or more sensors. The sensor data may be received from, for example and without limitation, a Global Navigation Satellite System (“GNSS”) sensor 1058 (e.g., Global Positioning System sensor), a RADAR sensor 1060, an ultrasonic sensor 1062, a LIDAR sensor 1064, an Inertial Measurement Unit (IMU) sensor 1066 (e.g., accelerometer, gyroscope, magnetic compass, magnetometer, etc.), a microphone 1096, a stereo camera 1068, a wide-angle camera 1070 (e.g., fisheye camera), an infrared camera 1072, a surround camera 1074 (e.g., 360-degree camera), a long-range and / or mid-range camera 1098, a speed sensor 1044 (e.g., for measuring the speed of vehicle 1000), a vibration sensor 1042, a steering sensor 1040, a brake sensor (e.g., as part of a brake sensor system 1046), and / or other sensor types.

[0090] One or more of the controllers 1036 may receive inputs (e.g., represented by input data) from the instrument cluster 1032 of vehicle 1000 and provide outputs (e.g., represented by output data, display data, etc.) via a Human Machine Interface (HMI) display 1034, an audible indicator, a speaker, and / or via other components of vehicle 1000. These outputs may include information such as vehicle speed, rate, time, map data (e.g., Figure 10C a high-definition (“HD”) map 1022), position data (e.g., the position of vehicle 1000 on a map, for example), direction, the positions of other vehicles (e.g., occupancy grids), and information about objects and object states as perceived by the controller 1036, and so on. For example, the HMI display 1034 may display information about the presence of one or more objects (e.g., street signs, warning signs, traffic light changes, etc.) and / or information about driving maneuvers that the vehicle has made, is making, or will make (e.g., changing lanes now, exiting 34B in two miles, etc.).

[0091] Vehicle 1000 also includes a network interface 1024, which may communicate over one or more networks using one or more wireless antennas 1026 and / or a modem. For example, network interface 1024 may be capable of communicating via Long Term Evolution (“LTE”), Wideband Code Division Multiple Access (“WCDMA”), Universal Mobile Telecommunications System (“UMTS”), Global System for Mobile Communications (“GSM”), IMT-CDMA Multi-Carrier (“CDMA2000”), etc. One or more wireless antennas 1026 may also enable communication between objects (such as vehicles, mobile devices, etc.) in an implementation environment using one or more local area networks of categories such as Bluetooth, Bluetooth Low Energy (“LE”), Z-Wave, ZigBee, etc. and / or one or more low-power wide area networks (LPWANs) of categories such as LoRaWAN, SigFox, etc.

[0092] Figure 10B For an example autonomous vehicle 1000 in accordance with some embodiments of the present disclosure for Figure 10A Examples of camera positions and fields of view of the example autonomous vehicle 1000. The cameras and respective fields of view are one example embodiment and are not intended to be limiting. For example, additional and / or alternative cameras may be included, and / or these cameras may be located at different positions on vehicle 1000.

[0093] The camera type for the cameras may include, but is not limited to, digital cameras that may be suitable for use with components and / or systems of vehicle 800. The cameras may operate under an Automotive Safety Integrity Level (ASIL) B and / or under another ASIL. The camera type may have any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc., depending on the embodiment. The cameras may be capable of using a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include a Red Clear Clear Clear (RCCC) color filter array, a Red Clear Clear Blue (RCCB) color filter array, a Red Blue Green Clear (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, clear pixel cameras such as those with RCCC, RCCB, and / or RBGC color filter arrays may be used in an effort to increase light sensitivity.

[0094] In some examples, one or more of the cameras can be used to perform Advanced Driver Assistance System (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-functional monocular camera can be installed to provide functions including lane departure warning, traffic sign assistance, and intelligent headlight control. One or more of the cameras (e.g., all cameras) can record and provide image data (e.g., video) simultaneously.

[0095] One or more of the cameras can be installed in mounting components such as custom-designed (three-dimensional (“3D”) printed) components to cut off stray light and reflections from within the vehicle (e.g., reflections from the instrument panel reflected in the windshield mirror) that may interfere with the image data capture ability of the camera. Regarding the wing mirror mounting component, the wing mirror assembly can be custom 3D printed such that the camera mounting plate matches the shape of the wing mirror. In some examples, one or more cameras can be integrated into the wing mirror. For side view cameras, one or more cameras can also be integrated into the four pillars at each corner of the cab.

[0096] A camera having a field of view that includes an environmental portion in front of the vehicle 1000 (e.g., a front camera) can be used for surround view to help identify the forward path and obstacles and, with the help of one or more controllers 1036 and / or a control SoC, assist in providing information crucial for generating an occupancy grid and / or determining a preferred vehicle path. The front camera can be used to perform many of the same ADAS functions as LIDAR, including emergency braking, pedestrian detection, and collision avoidance. The front camera can also be used for ADAS functions and systems, including lane departure warning (“LDW”), adaptive cruise control (“ACC”), and / or other functions such as traffic sign recognition.

[0097] A variety of cameras can be used in a front-mounted configuration, including, for example, monocular camera platforms that include complementary metal oxide semiconductor (“CMOS”) color imagers. Another example can be a wide-angle camera 1070, which can be used to sense objects (e.g., pedestrians, intersection traffic, or bicycles) entering the field of view from the periphery. Although Figure 10B only one wide-angle camera is illustrated, any number (including zero) of wide-angle cameras 1070 can be present on the vehicle 1000. Additionally, any number of long-range cameras 1098 (e.g., long-view stereo camera pairs) can be used for depth-based object detection, especially for objects for which a neural network has not been trained. The long-range cameras 1098 can also be used for object detection and classification and basic object tracking.

[0098] Any number of stereo cameras 1068 may also be included in a front-facing configuration. In at least one embodiment, one or more stereo cameras 1068 may include an integrated control unit that includes a scalable processing unit that may provide a multi-core microprocessor and programmable logic ("FPGA") with an integrated controller area network ("CAN") or Ethernet interface on a single chip. Such a unit may be used to generate a 3D map of the vehicle environment, including distance estimates for all points in the image. Alternatively, the stereo camera 1068 may include a compact stereo vision sensor that may include two camera lenses (one on the left and one on the right) and an image processing chip that may measure the distance from the vehicle to a target object and activate autonomous emergency braking and lane departure warning functions using the generated information (such as metadata). Other types of stereo cameras 1068 may be used in addition to or alternatively to those described herein.

[0099] Cameras having a field of view of an environmental portion that includes a side of the vehicle 1000 (such as side view cameras) may be used for surround view, providing information used to create and update an occupancy grid and generate side impact collision warnings. For example, surround cameras 1074 (such as the four surround cameras 1074 shown, for example, as Figure 10B in) may be disposed on the vehicle 1000. The surround cameras 1074 may include wide-angle cameras 1070, fish-eye cameras, 360-degree cameras, and / or the like. By way of example, four fish-eye cameras may be disposed on the front, rear, and sides of the vehicle. In an alternative arrangement, the vehicle may use three surround cameras 1074 (such as on the left, right, and rear), and one or more other cameras (such as a forward camera) may be utilized as a fourth surround camera.

[0100] Cameras having a field of view of an environmental portion that includes the rear of the vehicle 1000 (such as rear view cameras) may be used for assisting with parking, surround view, rear collision warnings, and creating and updating an occupancy grid. A variety of cameras may be used, including but not limited to cameras that are also suitable as front cameras as described herein (such as long-range and / or mid-range cameras 1098, stereo cameras 1068, infrared cameras 1072, etc.).

[0101] Figure 10C For use in accordance with some embodiments of the present disclosure Figure 10ABlock diagram of an example system architecture of an example autonomous vehicle 1000. It should be understood that this and other arrangements described herein are presented only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, function groupings, etc.) may be used in addition to or instead of those shown, and some elements may be omitted entirely. Further, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in combination with other components, and in any suitable combination and location. The various functions described herein as being performed by entities may be implemented by hardware, firmware, and / or software. For example, the various functions may be implemented by a processor executing instructions stored in memory.

[0102] Figure 10C Each of the components, features, and systems in vehicle 1000 is illustrated as being connected via bus 1002. Bus 1002 may include a Controller Area Network (CAN) data interface (alternatively referred to herein as the "CAN bus"). CAN may be a network within vehicle 1000 that aids in controlling various features and functions of vehicle 1000, such as driving features like brakes, acceleration, braking, steering, windshield wipers, etc. The CAN bus may be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., CAN ID). The CAN bus may be read to find steering wheel angle, ground speed, engine revolutions per minute (RPM), button positions, and / or other vehicle status indicators. The CAN bus may be ASIL B compliant.

[0103] Although bus 1002 is described herein as a CAN bus, this is not intended to be limiting. For example, in addition to or instead of the CAN bus, FlexRay and / or Ethernet may be used. Further, although bus 1002 is represented by a single line, this is not intended to be limiting. For example, any number of buses 1002 may exist, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 1002 may be used to perform different functions and / or may be used for redundancy. For example, a first bus 1002 may be used for collision avoidance functions, and a second bus 1002 may be used for driving control. In any example, each bus 1002 may communicate with any component of vehicle 1000, and two or more buses 1002 may communicate with the same component. In some examples, each SoC 1004, each controller 1036, and / or each computer within the vehicle may have access to the same input data (e.g., input from sensors of vehicle 1000) and may be connected to a common bus such as a CAN bus.

[0104] Vehicle 1000 may include one or more controllers 1036, such as those described herein with respect to Figure 10A the controllers described. The controller 1036 may be used for a variety of functions. The controller 1036 may be coupled to any other different components and systems of the vehicle 1000 and may be used for the control of the vehicle 1000, the artificial intelligence of the vehicle 1000, the infotainment for the vehicle 1000, and / or the like.

[0105] Vehicle 1000 may include one or more system-on-chips (SoCs) 1004. The SoC 1004 may include a CPU 1006, a GPU 1008, a processor 1010, a cache 1012, an accelerator 1014, a data store 1016, and / or other components and features not shown. In a variety of platforms and systems, the SoC 1004 may be used to control the vehicle 1000. For example, one or more SoCs 1004 may be combined with an HD map 1022 in a system (such as a system of the vehicle 1000), and the HD map may obtain map refreshes and / or updates from one or more servers (such as Figure 10D one or more servers 1078) via a network interface 1024.

[0106] The CPU 1006 may include a CPU cluster or a CPU complex (alternatively, referred to herein as "CCPLEX"). The CPU 1006 may include multiple cores and / or an L2 cache. For example, in some embodiments, the CPU 1006 may include eight cores in a coherent multiprocessor configuration. In some embodiments, the CPU 1006 may include four dual-core clusters, each with a dedicated L2 cache (such as a 2MB L2 cache). The CPU 1006 (such as CCPLEX) may be configured to support simultaneous cluster operation such that any combination of the clusters of the CPU 1006 can be active at any given time.

[0107] The CPU 1006 can implement power management capabilities including one or more of the following features: Each hardware block can automatically perform clock gating when idle to save dynamic power; Due to the execution of the WFI / WFE instructions, each core clock can be gated when the core is not actively executing instructions; Each core can perform power gating independently; When all cores perform clock gating or power gating, each core cluster can be clock gated independently; and / or When all cores perform power gating, each core cluster can be power gated independently. The CPU 1006 can further implement an enhanced algorithm for managing power states, where the allowed power states and the desired wake-up times are specified, and the hardware / microcode determines the optimal power state for the cores, clusters, and CCPLEX to enter. The processing cores can support a simplified power state entry sequence in software, and this work is offloaded to the microcode.

[0108] The GPU 1008 can include an integrated GPU (alternatively, referred to herein as "iGPU"). The GPU 1008 can be programmable and efficient for parallel workloads. In some examples, the GPU 1008 can use an enhanced tensor instruction set. The GPU 1008 can include one or more streaming microprocessors, where each streaming microprocessor can include an L1 cache (e.g., an L1 cache with at least 96 KB of storage capacity), and two or more of these streaming microprocessors can share an L2 cache (e.g., an L2 cache with 512 KB of storage capacity). In some embodiments, the GPU 1008 can include at least eight streaming microprocessors. The GPU 1008 can use a compute application programming interface (API). Additionally, the GPU 1008 can use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0109] In automotive and embedded use cases, the GPU 1008 can be power optimized for best performance. For example, the GPU 1008 can be fabricated on fin field-effect transistors (FinFETs). However, this is not intended to be limiting, and the GPU 1008 can be fabricated using other semiconductor manufacturing processes. Each streaming microprocessor can incorporate a number of mixed-precision processing cores divided into multiple blocks. For example and without limitation, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA tensor cores for deep learning matrix arithmetic, an L0 instruction cache, a warp scheduler, a dispatch unit, and / or a 64KB register file. Additionally, the streaming microprocessor can include separate parallel integer and floating-point data paths to enable efficient execution of workloads leveraging a mix of compute and addressing computations. The streaming microprocessor can include separate thread scheduling capabilities to allow for more fine-grained synchronization and cooperation between parallel threads. The streaming microprocessor can include a combined L1 data cache and shared memory unit to improve performance while simplifying programming.

[0110] The GPU 1008 can include, in some examples, high-bandwidth memory (HBM) that provides a peak memory bandwidth of approximately 900GB / s and / or a 16GB HBM2 memory subsystem. In some examples, in addition to or alternatively to HBM memory, synchronous graphics random access memory (SGRAM), such as fifth-generation graphics double data rate synchronous random access memory (GDDR5), can be used.

[0111] The GPU 1008 can include unified memory technology that includes access counters to allow memory pages to be more precisely migrated to the processors that most frequently access them, thereby improving the efficiency of the memory ranges shared between processors. In some examples, address translation service (ATS) support can be used to allow the GPU 1008 to directly access the CPU 1006 page tables. In such an example, when the GPU 1008 memory management unit (MMU) experiences a miss, an address translation request can be transmitted to the CPU 1006. In response, the CPU 1006 can look up the virtual-physical mapping for the address in its page table and transmit the translation back to the GPU 1008. In this way, the unified memory technology can allow a single unified virtual address space for the memory of both the CPU 1006 and the GPU 1008, thereby simplifying GPU 1008 programming and porting applications to the GPU 1008.

[0112] In addition, the GPU 1008 may include an access counter that can track how frequently the GPU 1008 accesses the memory of other processors. The access counter can help ensure that memory pages are moved to the physical memory of the processor that most frequently accesses those pages.

[0113] The SoC 1004 may include any number of caches 1012, including those described herein. For example, the cache 1012 may include an L3 cache (e.g., which is connected to both the CPU 1006 and the GPU 1008) that is available to both the CPU 1006 and the GPU 1008. The cache 1012 may include a write-back cache that can track the state of lines, for example, by using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). Depending on the embodiment, the L3 cache may include 4MB or more, but smaller cache sizes may also be used.

[0114] The SoC 1004 may include an arithmetic logic unit (ALU) that can be utilized in the processing of performing any of the various tasks or operations regarding the vehicle 1000 (such as processing a DNN). In addition, the SoC 1004 may include a floating-point unit (FPU) (or other math co-processor or digital co-processor type) for performing mathematical operations within the system. For example, the SoC 1004 may include one or more FPUs integrated as execution units within the CPU 1006 and / or the GPU 1008.

[0115] The SoC 1004 may include one or more accelerators 1014 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, the SoC 1004 may include a hardware accelerator cluster that may include optimized hardware accelerators and / or large on-chip memory. This large on-chip memory (e.g., 4MB SRAM) may enable the hardware accelerator cluster to accelerate neural networks and other computations. The hardware accelerator cluster may be used to supplement the GPU 1008 and offload some of the tasks of the GPU 1008 (e.g., freeing up more cycles of the GPU 1008 for performing other tasks). As an example, the accelerator 1014 may be used for targeted workloads that are stable enough to be easily accelerated (e.g., perception, convolutional neural network (CNN), etc.). As used herein, the term "CNN" may include all types of CNNs, including region-based or region convolutional neural networks (RCNN) and fast RCNN (e.g., for object detection).

[0116] The accelerator 1014 (e.g., a hardware accelerator cluster) may include a Deep Learning Accelerator (DLA). The DLA may include one or more Tensor Processing Units (TPUs) that can be configured to provide an additional one trillion operations per second for deep learning applications and inference. The TPU may be an accelerator configured to perform image processing functions (e.g., for CNN, RCNN, etc.) and optimized for performing image processing functions. The DLA may be further optimized for a specific set of neural network types and floating-point operations, and inference. The design of the DLA may provide higher performance per millimeter than a general-purpose GPU and far exceed the performance of a CPU. The TPU may perform several functions, including single-instance convolution functions, supporting INT8, INT16, and FP16 data types for both features and weights, and post-processor functions.

[0117] The DLA may execute neural networks, especially CNNs, quickly and efficiently for any of a variety of functions on processed or unprocessed data, such as, and without limitation: CNNs for object recognition and detection using data from a camera sensor; CNNs for distance estimation using data from a camera sensor; CNNs for emergency vehicle detection and identification and detection using data from a microphone; CNNs for face recognition and vehicle owner recognition using data from a camera sensor; and / or CNNs for security and / or safety-related events.

[0118] The DLA may perform any function of the GPU 1008, and by using an inference accelerator, for example, a designer may configure the DLA or the GPU 1008 for any function. For example, a designer may focus the processing and floating-point operations of a CNN on the DLA and leave other functions to the GPU 1008 and / or other accelerators 1014.

[0119] The accelerator 1014 (e.g., a hardware accelerator cluster) may include a Programmable Vision Accelerator (PVA), which may alternatively be referred to herein as a Computer Vision Accelerator. The PVA may be designed and configured to accelerate computer vision algorithms for Advanced Driver Assistance Systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. The PVA may provide a balance between performance and flexibility. For example, each PVA may include, for example and without limitation, any number of Reduced Instruction Set Computer (RISC) cores, Direct Memory Access (DMA), and / or any number of vector processors.

[0120] The RISC cores can interact with an image sensor (e.g., the image sensor of any camera described herein), an image signal processor, and / or the like. Each of these RISC cores can include any number of memories. Depending on the embodiment, the RISC cores can use any of several protocols. In some examples, the RISC cores can execute a real-time operating system (RTOS). The RISC cores can be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or storage devices. For example, the RISC cores can include an instruction cache and / or tightly coupled RAM.

[0121] The DMA can enable components of the PVA to access system memory independently of the CPU 1006. The DMA can support any number of features used to optimize the PVA, including but not limited to supporting multi-dimensional addressing and / or circular addressing. In some examples, the DMA can support addressing up to six or more dimensions, which can include block width, block height, block depth, horizontal block step, vertical block step, and / or depth step.

[0122] The vector processor can be a programmable processor that can be designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing capabilities. In some examples, the PVA can include a PVA core and two vector processing subsystem partitions. The PVA core can include a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem can operate as the main processing engine of the PVA and can include a vector processing unit (VPU), an instruction cache, and / or vector memory (e.g., VMEM). The VPU core can include a digital signal processor, such as, for example, a single instruction multiple data (SIMD), very long instruction word (VLIW) digital signal processor. The combination of SIMD and VLIW can enhance throughput and rate.

[0123] Each of the vector processors may include an instruction cache and may be coupled to dedicated memory. As a result, in some examples, each of the vector processors may be configured to execute independently of the other vector processors. In other examples, the vector processors included in a particular PVA may be configured to employ data parallelization. For example, in some embodiments, multiple vector processors included in a single PVA may execute the same computer vision algorithm, but on different regions of an image. In other examples, the vector processors included in a particular PVA may execute different computer vision algorithms simultaneously on the same image, or even execute different algorithms on sequential images or portions of an image. Among other things, any number of PVAs may be included in a hardware accelerator cluster, and any number of vector processors may be included in each of these PVAs. Additionally, the PVA may include additional error correction code (ECC) memory to enhance overall system security.

[0124] The accelerator 1014 (e.g., a hardware accelerator cluster) may include an on-chip computer vision network and SRAM to provide high-bandwidth, low-latency SRAM for the accelerator 1014. In some examples, the on-chip memory may include at least 4MB of SRAM consisting of, for example and without limitation, eight field-configurable memory blocks, which may be accessed by both the PVA and the DLA. Each pair of memory blocks may include an advanced peripheral bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory may be used. The PVA and the DLA may access the memory via a backbone that provides high-speed memory access to the PVA and the DLA. The backbone may include, for example using the APB, an on-chip computer vision network that interconnects the PVA and the DLA to the memory.

[0125] The on-chip computer vision network may include an interface that determines that both the PVA and the DLA provide ready and valid flags before transmitting any control flags / addresses / data. Such an interface may provide separate phases and separate channels for transmitting control flags / addresses / data, as well as burst communication for continuous data transmission. This type of interface may conform to the ISO 26262 or IEC 615010 standards, but other standards and protocols may also be used.

[0126] In some examples, SoC 1004 can include a real-time ray tracing hardware accelerator, such as that described in U.S. Patent Application No. 16 / 101,232, filed on August 10, 2018. The real-time ray tracing hardware accelerator can be used to quickly and efficiently determine the position and extent of objects (e.g., within a world model) to generate a real-time visualization simulation for RADAR signature interpretation, for sound propagation synthesis and / or analysis, for SONAR system simulation, for general wave propagation simulation, for comparison with LIDAR data for positioning and / or other functional purposes, and / or for other uses. In some embodiments, one or more tree traversal units (TTUs) can be used to perform one or more ray tracing related operations.

[0127] Accelerator 1014 (e.g., a hardware accelerator cluster) has a wide range of autonomous driving applications. The PVA can be a programmable vision accelerator that can be used in key processing stages in ADAS and autonomous vehicles. The capabilities of the PVA are a good match for algorithm domains that require predictable processing, low power, and low latency. In other words, the PVA performs well on semi-dense or dense regular computations, even on small data sets that require predictable runtimes with low latency and low power. Thus, in the context of a platform for autonomous vehicles, the PVA is designed to run classical computer vision algorithms because they are effective in object detection and integer math operations.

[0128] For example, according to one embodiment of the technology, the PVA is used to perform computer stereo vision. In some examples, an algorithm based on semi-global matching can be used, but this is not intended to be limiting. Many applications for level 3 - 5 autonomous driving require instantaneous motion estimation / stereo matching (e.g., structure from motion, pedestrian recognition, lane detection, etc.). The PVA can perform computer stereo vision functions on inputs from two monocular cameras.

[0129] In some examples, the PVA can be used to perform dense optical flow. Process raw RADAR data (e.g., using a 4D fast Fourier transform) to provide processed RADAR. In other examples, the PVA is used for time-of-flight depth processing, which, for example, processes raw time-of-flight data to provide processed time-of-flight data.

[0130] DLA can be used to run any type of network to enhance control and driving safety, including, for example, a neural network that outputs a confidence metric for each object detection. Such confidence values can be interpreted as probabilities or as providing a relative "weight" of each detection compared to other detections. The confidence value enables the system to make further decisions regarding which detections should be considered true positive detections rather than false positive detections. For example, the system can set a threshold for the confidence and consider only detections that exceed the threshold as true positive detections. In an automatic emergency braking (AEB) system, false positive detections can cause the vehicle to automatically perform emergency braking, which is clearly undesirable. Therefore, only the most confident detections should be considered as triggers for AEB. DLA can run a neural network for regressing confidence values. The neural network can take as its input at least some subset of parameters, such as bounding box dimensions, a ground plane estimate obtained (e.g., from another subsystem), the output of an inertial measurement unit (IMU) sensor 1066 related to the orientation and distance of the vehicle 1000, a 3D position estimate of an object obtained from the neural network and / or other sensors (such as a LIDAR sensor 1064 or a RADAR sensor 1060), etc.

[0131] The SoC 1004 can include one or more data stores 1016 (such as memory). The data store 1016 can be on-chip memory of the SoC 1004, which can store neural networks to be executed on the GPU and / or DLA. In some examples, for redundancy and safety, the data store 1016 can be large enough in capacity to store multiple instances of the neural network. The data store 1012 can include an L2 or L3 cache 1012. References to the data store 1016 can include references to memory associated with the PVA, DLA, and / or other accelerators 1014 as described herein.

[0132] The SoC 1004 may include one or more processors 1010 (e.g., embedded processors). The processor 1010 may include a boot and power management processor, which may be a dedicated processor and subsystem for handling boot power and management functions as well as security implementation related. The boot and power management processor may be part of the SoC 1004 boot sequence and may provide runtime power management services. The boot power and management processor may provide clock and voltage programming, assist system low power state transitions, manage the SoC 1004 heat and temperature sensors, and / or manage the SoC 1004 power state. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to temperature, and the SoC 1004 may use the ring oscillator to detect the temperature of the CPU 1006, GPU 1008, and / or accelerator 1014. If it is determined that the temperature exceeds a threshold, then the boot and power management processor may enter a temperature fault routine and place the SoC 1004 in a lower power state and / or place the vehicle 1000 in a driver safety stop mode (e.g., safely stop the vehicle 1000).

[0133] The processor 1010 may also include a set of embedded processors that can be used as an audio processing engine. The audio processing engine may be an audio subsystem that allows for full hardware support for multi-channel audio over multiple interfaces and a wide range of flexible audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core with a digital signal processor with dedicated RAM.

[0134] The processor 1010 may also include an always-on processor engine that can provide the necessary hardware features to support low power sensor management and wake-up use cases. The always-on processor engine may include a processor core, tightly coupled RAM, support peripherals (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0135] The processor 1010 may also include a security cluster engine that includes a dedicated processor subsystem for handling security management of automotive applications. The security cluster engine may include two or more processor cores, tightly coupled RAM, support peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In the security mode, the two or more cores may operate in a lockstep mode and act as a single core with comparison logic for detecting any differences between their operations.

[0136] The processor 1010 may also include a real-time camera engine that may include a dedicated processor subsystem for handling real-time camera management.

[0137] The processor 1010 may also include a high dynamic range flag processor, which may include an image flag processor, which is a hardware engine that is part of the camera processing pipeline.

[0138] The processor 1010 may include a video image compositor that may be a processing block (e.g., implemented on a microprocessor) that implements the video post-processing functions required for a video playback application to generate the final image for the player window. The video image compositor may perform lens distortion correction on the wide-angle camera 1070, the surround camera 1074, and / or the in-cab monitoring camera sensor. The in-cab monitoring camera sensor is preferably monitored by a neural network running on another instance of the advanced SoC, configured to identify in-cab events and respond accordingly. The in-cab system may perform lip reading to activate mobile phone services and make calls, dictate emails, change the vehicle destination, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. Certain functions are only available to the driver when the vehicle is operating in autonomous mode and are disabled otherwise.

[0139] The video image compositor may include enhanced temporal noise reduction for spatial and temporal noise reduction. For example, in the case of motion in the video, the noise reduction appropriately weights the spatial information, reducing the weight of the information provided by neighboring frames. In the case where the image or a portion of the image does not include motion, the temporal noise reduction performed by the video image compositor may use information from a previous image to reduce the noise in the current image.

[0140] The video image compositor may also be configured to perform stereo correction on input stereo lens frames. When the operating system desktop is in use and the GPU 1008 does not need to continuously render new surfaces, the video image compositor may be further used for user interface composition. Even when the GPU 1008 is powered on and active, performing 3D rendering, the video image compositor may be used to relieve the burden on the GPU 1008 to improve performance and responsiveness.

[0141] The SoC 1004 may also include a Mobile Industry Processor Interface (MIPI) camera serial interface, a high-speed interface, and / or a video input block for receiving video and inputs from cameras and may be used for camera and related pixel input functions. The SoC 1004 may also include an input / output controller that may be software-controlled and may be used to receive I / O flags that are not committed to a specific role.

[0142] SoC 1004 may also include a wide range of peripheral device interfaces to enable communication with peripheral devices, audio codecs, power management, and / or other devices. SoC 1004 can be used to process data from cameras (connected via Gigabit Multimedia Serial Link and Ethernet), sensors (such as LIDAR sensor 1064, RADAR sensor 1060, etc. that can be connected via Ethernet), data from bus 1002 (such as the speed of vehicle 1000, steering wheel position, etc.), and data from GNSS sensor 1058 (connected via Ethernet or CAN bus). SoC 1004 may also include dedicated high-performance large-capacity storage controllers, which may include their own DMA engines and can be used to free the CPU 1006 from routine data management tasks.

[0143] SoC 1004 can be an end-to-end platform with a flexible architecture that spans levels 3 - 5 of automation, thus providing an integrated functional safety architecture for a platform that utilizes and efficiently uses computer vision and ADAS technologies to achieve diversity and redundancy, along with deep learning tools to provide a flexible and reliable driving software stack. SoC 1004 can be faster, more reliable, and even more energy-efficient and space-efficient than conventional systems. For example, when combined with CPU 1006, GPU 1008, and data storage 1016, accelerator 1014 can provide a fast and efficient platform for level 3 - 5 autonomous vehicles.

[0144] Thus, this technology provides capabilities and functions that cannot be achieved by conventional systems. For example, computer vision algorithms can be executed on CPUs that can be configured using high-level programming languages such as the C programming language to perform various processing algorithms across a variety of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. In particular, many CPUs cannot execute complex object detection algorithms in real time, which is a requirement for in-vehicle ADAS applications and for practical level 3 - 5 autonomous vehicles.

[0145] In contrast to conventional systems, the technology described herein allows multiple neural networks to be executed simultaneously and / or sequentially by providing a CPU complex, a GPU complex, and a hardware accelerator cluster, and combining the results to achieve level 3 - 5 autonomous driving functions. For example, a CNN executed on a DLA or a dGPU (such as GPU 1020) can include text and word recognition, allowing a supercomputer to read and understand traffic signs, including signs for which the neural network has not been specifically trained. The DLA can also include a neural network capable of recognizing, interpreting, and providing semantic understanding of the signs and passing that semantic understanding to a path planning module running on the CPU complex.

[0146] As another example, as required for level 3, 4, or 5 driving, multiple neural networks can run simultaneously. For example, a warning sign consisting of "Caution: Flashing lights indicate icy conditions" together with the electric lights can be interpreted by several neural networks either independently or jointly. The sign itself can be recognized as a traffic sign by a first neural network deployed (e.g., a trained neural network), and the text "Flashing lights indicate icy conditions" can be interpreted by a second neural network deployed, which informs the vehicle's path planning software (preferably executed on the CPU complex) that when the flashing lights are detected, there are icy conditions. The flashing lights can be recognized by operating a third neural network deployed over multiple frames, which informs the vehicle's path planning software of the presence (or absence) of the flashing lights. All three neural networks can run simultaneously, for example, within the DLA and / or on the GPU 1008.

[0147] In some examples, the CNNs for face recognition and owner recognition can use data from the camera sensors to recognize the presence of an authorized driver and / or owner of the vehicle 1000. The processing engine always on the sensor can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and in the security mode, to disable the vehicle when the owner leaves the vehicle. In this way, the SoC 1004 provides security against theft and / or carjacking.

[0148] In another example, the CNN for emergency vehicle detection and recognition can use data from the microphone 1096 to detect and recognize an emergency vehicle siren. In contrast to conventional systems that use a general classifier to detect the siren and manually extract features, the SoC 1004 uses the CNN to classify environmental and urban sounds as well as visual data. In a preferred embodiment, the CNN running on the DLA is trained to recognize the relative closing rate of an emergency vehicle (e.g., by using the Doppler effect). The CNN can also be trained to recognize emergency vehicles specific to the local area in which the vehicle operates as recognized by the GNSS sensor 1058. Thus, for example, when operating in the EU, the CNN will seek to detect EU sirens, and when in the US, the CNN will seek to recognize only North American sirens. Once an emergency vehicle is detected, with the assistance of the ultrasonic sensor 1062, the control program can be used to execute emergency vehicle safety routines to slow down the vehicle, pull over to the side of the road, stop the vehicle, and / or let the vehicle idle until the emergency vehicle passes.

[0149] The vehicle may include a CPU 1018 (e.g., a discrete CPU or dCPU) that may be coupled to the SoC 1004 via a high-speed interconnect (e.g., PCIe). The CPU 1018 may include, for example, an X106 processor. The CPU 1018 may be used to perform any of a variety of functions, including, for example, arbitrating potentially inconsistent results between ADAS sensors and the SoC 1004, and / or monitoring the status and health of the controller 1036 and / or the infotainment SoC 1030.

[0150] The vehicle 1000 may include a GPU 1020 (e.g., a discrete GPU or dGPU) that may be coupled to the SoC 1004 via a high-speed interconnect (e.g., NVIDIA's NVLINK). The GPU 1020 may provide additional artificial intelligence capabilities, for example, by executing redundant and / or different neural networks, and may be used to train and / or update neural networks at least in part based on inputs (e.g., sensor data) from sensors of the vehicle 1000.

[0151] The vehicle 1000 may further include a network interface 1024, which may include one or more wireless antennas 1026 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). The network interface 1024 may be used to enable wireless connections to the cloud (e.g., to the server 1078 and / or other network devices), to other vehicles, and / or to computing devices (e.g., a passenger's client device) via the Internet. To communicate with other vehicles, a direct link may be established between the two vehicles, and / or an indirect link (e.g., across a network and via the Internet) may be established. The direct link may be provided using a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link may provide the vehicle 1000 with information about vehicles approaching the vehicle 1000 (e.g., vehicles in front of, to the side of, and / or behind the vehicle 1000). This function may be part of the cooperative adaptive cruise control function of the vehicle 1000.

[0152] The network interface 1024 may include an SoC that provides modulation and demodulation functions and enables the controller 1036 to communicate over a wireless network. The network interface 1024 may include a radio frequency front end for upconverting from baseband to radio frequency and downconverting from radio frequency to baseband. The frequency conversion may be performed by well-known processes and / or may be performed using a super-heterodyne process. In some examples, the radio frequency front end functions may be provided by a separate chip. The network interface may include wireless capabilities for communicating via LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0153] Vehicle 1000 may also include a data store 1028 that may include off-chip (e.g., outside of SoC 1004) storage devices. The data store 1028 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash memory, hard drives, and / or other components and / or devices that can store at least one bit of data.

[0154] Vehicle 1000 may also include a GNSS sensor 1058. The GNSS sensor 1058 (e.g., GPS, assisted GPS sensor, differential GPS (DGPS) sensor, etc.) is used to assist mapping, perception, occupancy grid generation, and / or path planning functions. Any number of GNSS sensors 1058 may be used, including, for example and without limitation, a GPS using a USB connector with an Ethernet to serial (RS-232) bridge.

[0155] Vehicle 1000 may also include a RADAR sensor 1060. The RADAR sensor 1060 may be used by vehicle 1000 for remote vehicle detection even in dark and / or adverse weather conditions. The RADAR functional safety level may be ASIL B. The RADAR sensor 1060 may use CAN and / or bus 1002 (e.g., to transmit data generated by the RADAR sensor 1060) for control as well as to access object tracking data, and in some examples access Ethernet to access raw data. A variety of RADAR sensor types may be used. For example and without limitation, the RADAR sensor 1060 may be suitable for front, rear, and side RADAR use. In some examples, a pulsed Doppler RADAR sensor is used.

[0156] The RADAR sensor 1060 may include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range side coverage, etc. In some examples, long-range RADAR may be used for adaptive cruise control functions. The long-range RADAR system may provide a wide field of view (e.g., within 250 m) achieved through two or more independent scans. The RADAR sensor 1060 may help distinguish between static and moving objects and may be used by the ADAS system for emergency braking assistance and forward collision warning. The long-range RADAR sensor may include a single station multi-mode RADAR with multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In an example with six antennas, the central four antennas may create a focused beam pattern that is designed to record the surroundings of vehicle 1000 at a higher rate with minimal traffic interference from adjacent lanes. The other two antennas may extend the field of view, making it possible to quickly detect vehicles entering or leaving the lane of vehicle 1000.

[0157] As an example, a mid-range RADAR system can include a range of up to 1060 m (front) or 80 m (rear) and a field of view of up to 42 degrees (front) or 1050 degrees (rear). A short-range RADAR system can include, but is not limited to, RADAR sensors designed to be mounted at both ends of the rear bumper. When mounted at both ends of the rear bumper, such a RADAR sensor system can create two beams that continuously monitor the rear and the blind spots beside the vehicle.

[0158] The short-range RADAR system can be used in an ADAS system for blind spot detection and / or lane change assistance.

[0159] Vehicle 1000 can also include ultrasonic sensors 1062. Ultrasonic sensors 1062 that can be placed in the front, rear, and / or sides of vehicle 1000 can be used for parking assistance and / or creating and updating occupancy grids. A variety of ultrasonic sensors 1062 can be used, and different ultrasonic sensors 1062 can be used for different detection ranges (e.g., 2.5 m, 4 m). Ultrasonic sensors 1062 can operate at ASIL B of the functional safety level.

[0160] Vehicle 1000 can include a LIDAR sensor 1064. The LIDAR sensor 1064 can be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LIDAR sensor 1064 can be at ASIL B of the functional safety level. In some examples, vehicle 1000 can include multiple LIDAR sensors 1064 (e.g., two, four, six, etc.) that can use Ethernet (e.g., to provide data to a gigabit Ethernet switch).

[0161] In some examples, the LIDAR sensor 1064 may be able to provide a list of objects and their distances for a 360-degree field of view. Commercially available LIDAR sensors 1064 can have, for example, an advertised range of approximately 1000 m, an accuracy of 2 cm - 3 cm, and support a 1000 Mbps Ethernet connection. In some examples, one or more non-protruding LIDAR sensors 1064 can be used. In such examples, the LIDAR sensor 1064 can be implemented as a small device that can be embedded in the front, rear, sides, and / or corners of vehicle 1000. In such examples, the LIDAR sensor 1064 can provide a field of view of up to 120 degrees horizontally and 35 degrees vertically, with a range of 200 m, even for low-reflectivity objects. The front-mounted LIDAR sensor 1064 can be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0162] In some examples, LIDAR technologies such as 3D flash LIDAR can also be used. 3D flash LIDAR uses the flash of a laser as the emission source to illuminate the vehicle's surroundings up to about 200m. The flash LIDAR unit includes a receiver that records the laser pulse transmission time and the reflected light on each pixel, which in turn corresponds to the range from the vehicle to the object. Flash LIDAR can allow for the generation of highly accurate and distortion-free images of the surroundings using each laser flash. In some examples, four flash LIDAR sensors can be deployed, one on each side of the vehicle 1000. Available 3D flash LIDAR systems include solid-state 3D staring array LIDAR cameras (e.g., non-scanning LIDAR devices) that have no moving parts other than a fan. The flash LIDAR device can use Class I (eye-safe) laser pulses of 5 nanoseconds per frame and can capture the reflected laser in the form of 3D range point clouds and co-registered intensity data. By using flash LIDAR and since flash LIDAR is a solid-state device with no moving parts, the LIDAR sensor 1064 can be less susceptible to motion blur, vibration, and / or shock.

[0163] The vehicle can also include an IMU sensor 1066. In some examples, the IMU sensor 1066 can be located at the center of the rear axle of the vehicle 1000. The IMU sensor 1066 can include, for example and without limitation, accelerometers, magnetometers, gyroscopes, magnetic compasses, and / or other sensor types. In some examples, such as in six-axis applications, the IMU sensor 1066 can include an accelerometer and a gyroscope, while in nine-axis applications, the IMU sensor 1066 can include an accelerometer, a gyroscope, and a magnetometer.

[0164] In some embodiments, the IMU sensor 1066 can be implemented as a miniature high-performance GPS-aided inertial navigation system (GPS / INS) that combines microelectromechanical system (MEMS) inertial sensors, a high-sensitivity GPS receiver, and advanced Kalman filtering algorithms to provide estimates of position, velocity, and attitude. Thus, in some examples, the IMU sensor 1066 can enable the vehicle 1000 to estimate the heading without input from a magnetic sensor by directly observing the speed changes from the GPS to the IMU sensor 1066 and correlating them. In some examples, the IMU sensor 1066 and the GNSS sensor 1058 can be integrated into a single unit.

[0165] The vehicle can include a microphone 1096 placed in and / or around the vehicle 1000. Among other things, the microphone 1096 can be used for emergency vehicle detection and identification.

[0166] The vehicle may also include any number of camera types, including a stereo camera 1068, a wide-angle camera 1070, an infrared camera 1072, a surround camera 1074, a long-range and / or mid-range camera 1098, and / or other camera types. These cameras can be used to capture image data around the entire periphery of the vehicle 1000. The camera types used depend on the embodiment and the requirements of the vehicle 1000, and any combination of camera types can be used to provide the necessary coverage around the vehicle 1000. Additionally, the number of cameras can vary according to the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. As an example and without limitation, these cameras can support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet. Each of the cameras is described in more detail herein with respect to Figure 10A and Figure 10B is described in more detail.

[0167] The vehicle 1000 may also include a vibration sensor 1042. The vibration sensor 1042 can measure the vibration of components of the vehicle such as axles. For example, a change in vibration can indicate a change in the road surface. In another example, when two or more vibration sensors 1042 are used, the difference between the vibrations can be used to determine the friction or slip of the road surface (e.g., when there is a vibration difference between a powered driving axle and a freely rotating axle).

[0168] The vehicle 1000 may include an ADAS system 1038. In some examples, the ADAS system 1038 may include a SoC. The ADAS system 1038 may include autonomous / adaptive / auto cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keeping assist (LKA), blind spot warning (BSW), rear cross-traffic warning (RCTW), collision warning system (CWS), lane centering (LC), and / or other features and functions.

[0169] The ACC system can use RADAR sensors 1060, LIDAR sensors 1064, and / or cameras. The ACC system can include longitudinal ACC and / or lateral ACC. Longitudinal ACC monitors and controls the distance to the vehicle immediately in front of the vehicle 1000 and automatically adjusts the vehicle speed to maintain a safe distance from the vehicle in front. Lateral ACC performs distance keeping and, when necessary, advises the vehicle 1000 to change lanes. Lateral ACC is related to other ADAS applications such as LCA and CWS.

[0170] The CACC uses information from other vehicles, which can be received indirectly from other vehicles via the network interface 1024 and / or the wireless antenna 1026 via a wireless link or through a network connection (e.g., via the Internet). The direct link can be provided by a vehicle-to-vehicle (V2V) communication link, while the indirect link can be an infrastructure-to-vehicle (I2V) communication link. Generally, the V2V communication concept provides information about the immediately preceding vehicle (e.g., the vehicle immediately in front of vehicle 1000 and in the same lane as it), while the I2V communication concept provides information about traffic further ahead. The CACC system can include either or both of the I2V and V2V information sources. Given information about the vehicle in front of vehicle 1000, the CACC can be more reliable, and it has the potential to improve the smoothness of traffic flow and reduce road congestion.

[0171] The FCW system is designed to alert the driver of a hazard so that the driver can take corrective action. The FCW system uses a front camera and / or RADAR sensor 1060 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating component. The FCW system can provide warnings in the form of, for example, audible, visual warnings, vibrations, and / or rapid braking pulses.

[0172] The AEB system detects an impending frontal collision with another vehicle or other object and can automatically apply the brakes if the driver does not take corrective action within specified time or distance parameters. The AEB system can use a front camera and / or RADAR sensor 1060 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. When the AEB system detects a hazard, it typically first alerts the driver to take corrective action to avoid the collision, and if the driver does not take corrective action, then the AEB system can automatically apply the brakes in an effort to prevent or at least mitigate the impact of the predicted collision. The AEB system can include technologies such as dynamic brake support and / or collision imminent braking.

[0173] The LDW system provides visual, audible, and / or tactile warnings such as steering wheel or seat vibrations to alert the driver when vehicle 1000 crosses a lane marking. The LDW system is not activated when the driver indicates an intentional lane departure by activating a turn signal. The LDW system can use a front-side-facing camera coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating component.

[0174] The LKA system is a variant of the LDW system. If the vehicle 1000 starts to leave the lane, then the LKA system provides a steering input or braking to correct the vehicle 1000.

[0175] The BSW system detects and warns the driver of vehicles in the vehicle's blind spot. The BSW system can provide visual, audible, and / or tactile alerts to indicate that merging or changing lanes is unsafe. The system can provide additional warnings when the driver uses a turn signal. The BSW system can use a rear-facing camera and / or RADAR sensor 1060 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating component.

[0176] The RCTW system can provide visual, audible, and / or tactile notifications when an object is detected outside the rear camera range while the vehicle 1000 is reversing. Some RCTW systems include AEB to ensure that the vehicle brakes are applied to avoid a crash. The RCTW system can use one or more rear RADAR sensors 1060 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating component.

[0177] Conventional ADAS systems may be prone to false positive results, which can be annoying and distracting to the driver, but typically not catastrophic because the ADAS system alerts the driver and allows the driver to decide whether a safe condition truly exists and act accordingly. However, in an autonomous vehicle 1000, in the case of conflicting results, the vehicle 1000 itself must decide whether to heed the results from the main computer or an auxiliary computer (e.g., the first controller 1036 or the second controller 1036). For example, in some embodiments, the ADAS system 1038 can be a backup and / or auxiliary computer for providing perception information to a redundant computer sanity module. The redundant computer sanity monitor can run redundant and diverse software on hardware components to detect faults in perception and dynamic driving tasks. The output from the ADAS system 1038 can be provided to the supervisory MCU. If the outputs from the main computer and the auxiliary computer conflict, then the supervisory MCU must determine how to reconcile the conflict to ensure safe operation.

[0178] In some examples, the host computer may be configured to provide a confidence score to the supervisory MCU indicating the host computer's confidence in the selected result. If the confidence score exceeds a threshold, then the supervisory MCU may follow the direction of the host computer regardless of whether the secondary computer provides conflicting or inconsistent results. In cases where the confidence score does not meet the threshold and where the host computer and the secondary computer indicate different results (e.g., conflict), the supervisory MCU may arbitrate between these computers to determine an appropriate result.

[0179] The supervisory MCU may be configured to run a neural network that is trained and configured to determine conditions under which the secondary computer provides a false alarm, at least in part based on outputs from the host computer and the secondary computer. Thus, the neural network in the supervisory MCU can learn when the output of the secondary computer can be trusted and when it cannot. For example, when the secondary computer is a RADAR-based FCW system, the neural network in the supervisory MCU can learn when the FCW system is identifying metal objects that are not in fact dangerous, such as a drainage grate or manhole cover that triggers an alarm. Similarly, when the secondary computer is a camera-based LDW system, the neural network in the supervisory MCU can learn to ignore the LDW when a cyclist or pedestrian is present and lane departure is actually the safest strategy. In embodiments that include a neural network running on the supervisory MCU, the supervisory MCU may include at least one of a DLA or a GPU suitable for running the neural network with associated memory. In a preferred embodiment, the supervisory MCU may include components of SoC 1004 and / or be included as a component of SoC 1004.

[0180] In other examples, the ADAS system 1038 may include a secondary computer that performs ADAS functions using traditional computer vision rules. In this way, the secondary computer may use classical computer vision rules (if-then), and the presence of a neural network in the supervisory MCU can improve reliability, safety, and performance. For example, the diverse implementations and intentional non-identity make the overall system more fault-tolerant, especially for failures caused by software (or software-hardware interface) functions. For example, if there is a software vulnerability or error in the software running on the host computer and the non-identical software code running on the secondary computer provides the same overall result, then the supervisory MCU can be more confident that the overall result is correct and that the vulnerability in the software or hardware on the host computer does not cause a substantial error.

[0181] In some examples, the output of the ADAS system 1038 can be fed to the perception block of the main computer and / or the dynamic driving task block of the main computer. For example, if the ADAS system 1038 indicates a forward collision warning due to an object being immediately in front, then the perception block can use that information when identifying the object. In other examples, the auxiliary computer can have its own neural network, which is trained and thus reduces the risk of false positives as described herein.

[0182] Vehicle 1000 can also include an infotainment SoC 1030 (e.g., in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system can not be an SoC and can include two or more discrete components. The infotainment SoC 1030 can include a combination of hardware and software that can be used to provide audio (e.g., music, personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., LTE, WiFi, etc.), and / or information services (e.g., navigation system, rear parking assistance, radio data system, vehicle-related information such as fuel level, total distance covered, brake fuel level, oil level, door open / close, air filter information, etc.) to vehicle 1000. For example, the infotainment SoC 1030 can include a radio, disc player, navigation system, video player, USB and Bluetooth connectivity, in-vehicle computer, in-vehicle entertainment, WiFi, steering wheel audio controls, hands-free voice controls, head-up display (HUD), HMI display 1034, telematics device, control panel (e.g., for controlling various components, features, and / or systems, and / or interacting with them), and / or other components. The infotainment SoC 1030 can further be used to provide information (e.g., visual and / or auditory) to the users of the vehicle, such as information from the ADAS system 1038, autonomous driving information such as planned vehicle maneuvers, trajectories, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0183] The infotainment SoC 1030 can include GPU functionality. The infotainment SoC 1030 can communicate with other devices, systems, and / or components of vehicle 1000 via a bus 1002 (e.g., CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 1030 can be coupled to the supervisory MCU such that in the event of a failure of the main controller 1036 (e.g., the main and / or backup computer of vehicle 1000), the GPU of the infotainment system can perform some autonomous driving functions. In such examples, the infotainment SoC 1030 can place vehicle 1000 in the driver safe stop mode as described herein.

[0184] Vehicle 1000 may further include an instrument cluster 1032 (such as a digital instrument panel, an electronic instrument cluster, a digital instrument surface panel, etc.). The instrument cluster 1032 may include a controller and / or a supercomputer (such as a discrete controller or supercomputer). The instrument cluster 1032 may include a set of instruments, such as a speedometer, fuel level, oil pressure, tachometer, odometer, turn indicator, shift position indicator, seat belt warning light, parking brake warning light, engine malfunction light, airbag (SRS) system information, lighting controls, safety system controls, navigation information, and so on. In some examples, information may be displayed and / or shared between the infotainment SoC 1030 and the instrument cluster 1032. In other words, the instrument cluster 1032 may be included as part of the infotainment SoC 1030, or vice versa.

[0185] Figure 10D A system schematic diagram for communication between a cloud-based server and Figure 10A an example autonomous vehicle 1000 according to some embodiments of the present disclosure. The system 1076 may include a server 1078, a network 1090, and vehicles including the vehicle 1000. The server 1078 may include multiple GPUs 1084(A)-1084(H) (collectively referred to herein as GPUs 1084), PCIe switches 1082(A)-1082(H) (collectively referred to herein as PCIe switches 1082), and / or CPUs 1080(A)-1080(B) (collectively referred to herein as CPUs 1080). The GPUs 1084, CPUs 1080, and PCIe switches may be interconnected by high-speed interconnects such as, for example and without limitation, an NVLink interface 1088 developed by NVIDIA and / or a PCIe connection 1086. In some examples, the GPUs 1084 are connected via NVLink and / or an NVSwitch SoC, and the GPUs 1084 and the PCIe switches 1082 are connected via a PCIe interconnect. Although eight GPUs 1084, two CPUs 1080, and two PCIe switches are illustrated, this is not intended to be limiting. Depending on the embodiment, each of the servers 1078 may include any number of GPUs 1084, CPUs 1080, and / or PCIe switches. For example, each of the servers 1078 may include eight, sixteen, thirty-two, and / or more GPUs 1084.

[0186] Server 1078 can receive image data via network 1090 and from a vehicle, the image data representing an image showing an unexpected or changed road condition such as a recently started road work. Server 1078 can transmit neural network 1092, updated neural network 1092, and / or map information 1094, including information about traffic and road conditions, via network 1090 and to the vehicle. Updates to the map information 1094 can include updates to the HD map 1022, such as information about construction sites, potholes, curves, floods, or other obstacles. In some examples, the neural network 1092, updated neural network 1092, and / or map information 1094 can be represented and / or generated based on data received from new training and / or from any number of vehicles in the environment and / or experience from training performed at a data center (e.g., using server 1078 and / or other servers).

[0187] Server 1078 can be used to train a machine learning model (e.g., a neural network) based on training data. The training data can be generated by vehicles, and / or can be generated in simulation (e.g., using a game engine). In some examples, the training data is labeled (e.g., in cases where the neural network benefits from supervised learning) and / or undergoes other preprocessing, while in other examples, the training data is not labeled and / or preprocessed (e.g., in cases where the neural network does not require supervised learning). Training can be performed according to any one or more categories of machine learning techniques, including but not limited to categories such as: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, joint learning, transfer learning, feature learning (including principal component and clustering analysis), multilinear subspace learning, manifold learning, representation learning (including alternative dictionary learning), rule-based machine learning, anomaly detection, and any variants or combinations thereof. Once the machine learning model is trained, the machine learning model can be used by the vehicle (e.g., transmitted to the vehicle via network 1090), and / or the machine learning model can be used by server 1078 to remotely monitor the vehicle.

[0188] In some examples, server 1078 can receive data from a vehicle and apply the data to the latest real-time neural network for real-time intelligent inference. Server 1078 can include a deep learning supercomputer powered by GPU 1084 and / or a dedicated AI computer, such as DGX and DGX Station machines developed by NVIDIA. However, in some examples, server 1078 can include a deep learning infrastructure of a data center powered only by a CPU.

[0189] The deep learning infrastructure of server 1078 may be capable of fast real-time inference and can use this ability to evaluate and verify the health of the processors, software, and / or associated hardware in vehicle 1000. For example, the deep learning infrastructure may receive periodic updates from vehicle 1000, such as a sequence of images and / or objects located in the image sequence that vehicle 1000 has located (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure may run its own neural network to identify the objects and compare them with the objects identified by vehicle 1000. If the results do not match and the infrastructure concludes that the AI in vehicle 1000 has failed, then server 1078 may transmit a flag to vehicle 1000 instructing the fail-safe computer in vehicle 1000 to take control, notify the passengers, and complete a safe parking operation.

[0190] For inference, server 1078 may include a GPU 1084 and one or more programmable inference accelerators (e.g., NVIDIA's TensorRT). The combination of a GPU-powered server and inference acceleration may enable real-time response. In other examples, such as when performance is less critical, a CPU, FPGA, and other processor-powered servers may be used for inference.

[0191] Example computing device

[0192] Figure 11 is a block diagram of an example computing device 1100 suitable for implementing some embodiments of the present disclosure. Computing device 1100 may include an interconnect system 1102 that directly or indirectly couples the following devices: a memory 1104, one or more central processing units (CPUs) 1106, one or more graphics processing units (GPUs) 1108, a communication interface 1110, input / output (I / O) ports 1112, input / output components 1114, a power supply 1116, one or more presentation components 1118 (e.g., one or more displays), and one or more logic units 1120. In at least one embodiment, one or more computing devices 1100 may include one or more virtual machines (VMs), and / or any of its components may include virtual components (e.g., virtual hardware components). For non-limiting examples, one or more of the GPUs 1108 may include one or more vGPUs, one or more of the CPUs 1106 may include one or more vCPUs, and / or one or more of the logic units 1120 may include one or more virtual logic units. Thus, one or more computing devices 1100 may include discrete components (e.g., full GPUs dedicated to computing device 1100), virtual components (e.g., a portion of a GPU dedicated to computing device 1100), or a combination thereof.

[0193] Although Figure 11 each of the boxes of is shown as being connected via a circuit through the interconnect system 1102, this is not intended to be limiting and is for clarity only. For example, in some embodiments, a presentation component 1118 (such as a display device) may be considered an I / O component 1114 (e.g., if the display is a touch screen). As another example, the CPU 1106 and / or GPU 1108 may include memory (e.g., the memory 1104 may represent a storage device in addition to the memory of the GPU 1108, the CPU 1106, and / or other components). In other words, Figure 11 the computing devices of are merely illustrative. No distinction is made between categories such as "workstation", "server", "laptop computer", "desktop computer", "tablet computer", "client device", "mobile device", "handheld device", "gaming console", "electronic control unit (ECU)", "virtual reality system", and / or other device or system types, as all are considered within the scope of Figure 11 the computing devices of.

[0194] The interconnect system 1102 may represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnect system 1102 may include one or more bus or link types, such as an Industry Standard Architecture (ISA) bus, an Extended Industry Standard Architecture (EISA) bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe) bus, and / or another type of bus or link. In some embodiments, there are direct connections between components. As an example, the CPU 1106 may be directly connected to the memory 1104. Further, the CPU 1106 may be directly connected to the GPU 1108. In cases where there are direct or point-to-point connections between components, the interconnect system 1102 may include a PCIe link to perform the connection. In these examples, a PCI bus need not be included in the computing device 1100.

[0195] The memory 1104 may include any of a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by the computing device 1100. Computer-readable media can include volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, computer-readable media may include computer storage media and communication media.

[0196] Computer storage media can include volatile and non-volatile media and / or removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 1104 can store computer-readable instructions (e.g., representing one or more programs and / or one or more program elements such as an operating system). Computer storage media can include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic tape cartridges, magnetic tape, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by computing device 1100. As used herein, computer storage media does not include signals per se.

[0197] Computer storage media can embody computer-readable instructions, data structures, program modules, and / or other data types in a modulated data signal such as a carrier wave or other transport mechanism, and include any information delivery medium. The term "modulated data signal" can refer to a signal that sets or changes one or more of its characteristics in such a manner as to encode information in the signal. By way of example, and not limitation, computer storage media can include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of any of the above should also be included within the scope of computer-readable media.

[0198] CPU 1106 can be configured to execute at least some of the computer-readable instructions to control one or more components of computing device 1100 to perform one or more of the methods and / or processes described herein. Each of CPU 1106 can include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of concurrently handling numerous software threads. CPU 1106 can include any type of processor and can include different types of processors depending on the type of computing device 1100 being implemented (e.g., a processor with fewer cores for a mobile device and a processor with more cores for a server). For example, depending on the type of computing device 1100, the processor can be an advanced RISC machine (ARM) processor implemented using reduced instruction set computing (RISC) or an x86 processor implemented using complex instruction set computing (CISC). In addition to one or more microprocessors or secondary coprocessors such as a math coprocessor, computing device 1100 can also include one or more CPU 1106.

[0199] In addition to or instead of one or more CPUs 1106, one or more GPUs 1108 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1100 to perform one or more of the methods and / or processes described herein. One or more of the GPUs 1108 may be an integrated GPU (e.g., one of the CPUs 1106) and / or one or more of the GPUs 1108 may be a discrete GPU. In an embodiment, one or more of the GPUs 1108 may be a coprocessor of one or more of the CPUs 1106. The GPUs 1108 may be used by the computing device 1100 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, the GPUs 1108 may be used for general-purpose computing on GPUs (GPGPU). The GPUs 1108 may include hundreds or thousands of cores capable of handling hundreds or thousands of software threads simultaneously. The GPUs 1108 may generate pixel data of an output image in response to a rendering command (e.g., a rendering command received from the CPU 1106 via a host interface). The GPUs 1108 may include graphics memory (e.g., display memory) for storing pixel data or any other suitable data (e.g., GPGPU data). The display memory may be included as part of the memory 1104. The GPUs 1108 may include two or more GPUs operating in parallel (e.g., via a link). The link may directly connect the GPUs (e.g., using NVLINK) or may connect the GPUs through a switch (e.g., using NVSwitch). When combined, each GPU 1108 may generate pixel data or GPGPU data for different parts of the output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory or may share memory with other GPUs.

[0200] In addition to or in lieu of the CPU 1106 and / or GPU 1108, the logic unit 1120 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1100 to perform one or more of the methods and / or processes described herein. In an embodiment, one or more CPUs 1106, one or more GPUs 1108, and / or one or more logic units 1120 may perform any combination of methods, processes, and / or portions thereof discretely or jointly. One or more of the logic units 1120 may be part of and / or integrated in one or more of the CPUs 1106 and / or GPUs 1108 and / or one or more of the logic units 1120 may be discrete components or otherwise external to the CPUs 1106 and / or GPUs 1108. In an embodiment, one or more of the logic units 1120 may be a coprocessor of one or more of the CPUs 1106 and / or GPUs 1108.

[0201] Examples of the logic unit 1120 include one or more processing cores and / or their components, such as a data processing unit (DPU), a tensor core (TC), a tensor processing unit (TPU), a pixel vision core (PVC), a vision processing unit (VPU), a graphics processing cluster (GPC), a texture processing cluster (TPC), a streaming multiprocessor (SM), a tree traversal unit (TTU), an artificial intelligence accelerator (AIA), a deep learning accelerator (DLA), an arithmetic logic unit (ALU), an application specific integrated circuit (ASIC), a floating point unit (FPU), an input / output (I / O) element, a peripheral component interconnect (PCI) or a peripheral component interconnect express (PCIe) element, etc.

[0202] The communication interface 1110 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 1100 to communicate with other computing devices via an electronic communication network, including wired and / or wireless communication. The communication interface 1110 may include components and functionality for enabling communication over any of a plurality of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., over Ethernet or InfiniBand communication), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, one or more logic units 1120 and / or the communication interface 1110 may include one or more data processing units (DPUs) to directly send data received over the network and / or via the interconnect system 1102 to one or more GPUs 1108 (e.g., of the memory).

[0203] The I / O port 1112 may enable the computing device 1100 to be logically coupled to other devices including I / O components 1114, one or more presentation components 1118, and / or other components, some of which may be built into (e.g., integrated in) the computing device 1100. Illustrative I / O components 1114 include a microphone, a mouse, a keyboard, a joystick, a gamepad, a game controller, a satellite dish, a scanner, a printer, a wireless device, and the like. The I / O components 1114 may provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by a user. In some cases, the input may be transmitted to appropriate network elements for further processing. The NUI may implement any combination of speech recognition, stylus recognition, face recognition, biometric recognition, on-screen and near-screen gesture recognition, air gestures, head and eye tracking, and touch recognition (as described in more detail below) associated with the display of the computing device 1100. The computing device 1100 may include a depth camera for gesture detection and recognition, such as a stereoscopic camera system, an infrared camera system, an RGB camera system, touchscreen technology, and combinations thereof. Additionally, the computing device 1100 may include an accelerometer or gyroscope (e.g., as part of an inertial measurement unit (IMU)) that enables the detection of motion. In some examples, the computing device 1100 may use the output of the accelerometer or gyroscope to render immersive augmented reality or virtual reality.

[0204] The power supply 1116 may include a hardwired power supply, a battery power supply, or a combination thereof. The power supply 1116 may provide power to the computing device 1100 to enable the components of the computing device 1100 to operate.

[0205] The presentation component 1118 may include a display (e.g., a monitor, a touch screen, a television screen, a head-up display (HUD), other display types, or a combination thereof), a speaker, and / or other presentation components. The presentation component 1118 may receive data from other components (e.g., the GPU 1108, the CPU 1106, etc.) and output the data (e.g., as an image, a video, a sound, etc.).

[0206] Example data center

[0207] Figure 12 An example data center 1200 that may be used in at least one embodiment of the present disclosure is shown. The data center 1200 may include a data center infrastructure layer 1210, a framework layer 1220, a software layer 1230, and / or an application layer 1240.

[0208] As Figure 12 shown, the data center infrastructure layer 1210 may include a resource coordinator 1210, grouped computing resources 1214, and node computing resources ("node C.R.s") 1216(1)-1216(N), where "N" represents any whole positive integer. In at least one embodiment, the node C.R.s 1216(1)-1216(N) may include, but are not limited to, any number of central processing units ("CPUs") or other processors (including DPUs, accelerators, field programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output ("NW I / O") devices, network switches, virtual machines ("VMs"), power modules, and / or cooling modules, etc. In some embodiments, one or more of the node C.R.s 1216(1)-1216(N) may correspond to a server having one or more of the above computing resources. Additionally, in some embodiments, the node C.R.s 1216(1)-12161(N) may include one or more virtual components, such as vGPUs, vCPUs, etc., and / or one or more of the node C.R.s 1216(1)-1216(N) may correspond to a virtual machine (VM).

[0209] In at least one embodiment, the grouped computing resources 1214 may include separate groupings of node C.R.s 1216 housed within one or more racks (not shown), or many racks within data centers located at different geographical locations (also not shown). Separate groupings of node C.R.s 1216 within the grouped computing resources 1214 may include grouped computing, networking, memory, or storage resources that may be configured or allocated to support one or more workloads. In at least one embodiment, several node C.R.s 1216 including CPUs, GPUs, and / or other processors may be grouped within one or more racks to provide computing resources to support one or more workloads. One or more racks may also include any combination of any number of power modules, cooling modules, and / or network switches.

[0210] The resource coordinator 1222 may configure or otherwise control one or more node C.R.s 1216(1)-1216(N) and / or the grouped computing resources 1214. In at least one embodiment, the resource coordinator 1222 may include a software design infrastructure (“SDI”) management entity for the data center 1200. The resource coordinator 1222 may include hardware, software, or some combination thereof.

[0211] In at least one embodiment, as Figure 12 shown, the framework layer 1220 may include a job scheduler 1232, a configuration manager 1234, a resource manager 1236, and / or a distributed file system 1238. The framework layer 1220 may include a framework for software 1232 of the support software layer 1230 and / or one or more applications 1242 of the application layer 1240. The software 1232 or the application 1242 may respectively include network-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. The framework layer 1220 may be, but is not limited to, a free and open-source software web application framework (such as Apache Spark) that may utilize the distributed file system 1238 for large-scale data processing (e.g., “big data”) TM(hereinafter referred to as "Spark"). In at least one embodiment, the job scheduler 1232 may include a Spark driver to facilitate scheduling of workloads supported by different layers of the data center 1200. The configuration manager 1234 may be able to configure different layers, such as the software layer 1230 and the framework layer 1220 (which includes Spark and the distributed file system 1238 for supporting large-scale data processing). The resource manager 1236 may be able to manage the clustered or grouped computing resources mapped to or allocated for supporting the distributed file system 1238 and the job scheduler 1232. In at least one embodiment, the clustered or grouped computing resources may include the grouped computing resources 1214 in the data center infrastructure layer 1210. The resource manager 1236 may coordinate with the resource coordinator 1210 to manage these mapped or allocated computing resources.

[0212] In at least one embodiment, the software 1232 included in the software layer 1230 may include software used by at least a portion of the node C.R.s 1216(1)-1216(N), the grouped computing resources 1214, and / or the distributed file system 1238 of the framework layer 1220. One or more types of software may include, but are not limited to, Internet web search software, email virus scanning software, database software, and streaming video content software.

[0213] In at least one embodiment, the applications 1242 included in the application layer 1240 may include one or more types of applications used by at least a portion of the node C.R.s 1216(1)-1216(N), the grouped computing resources 1214, and / or the distributed file system 1238 of the framework layer 1220. One or more types of applications may include, but are not limited to, any number of genomic applications, cognitive computing, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments.

[0214] In at least one embodiment, any one of the configuration manager 1234, the resource manager 1236, and the resource coordinator 1210 may implement any number and type of self-modifying actions based on any amount and type of data obtained in any technically feasible manner. The self-modifying actions may free the data center operator of the data center 1200 from making potentially suboptimal configuration decisions and may avoid underutilization and / or suboptimal execution portions of the data center.

[0215] According to one or more embodiments described herein, data center 1200 may include tools, services, software, or other resources to train one or more machine learning models or to use one or more machine learning models to predict or infer information. For example, one or more machine learning models may be trained by computing weight parameters according to a neural network architecture by using the software and / or computing resources described above with respect to data center 1200. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks may be used to infer or predict information by using the resources described above with respect to data center 1200 with weight parameters computed by using one or more training techniques (such as but not limited to those described herein).

[0216] In at least one embodiment, data center 1200 may use a CPU, application specific integrated circuit (ASIC), GPU, FPGA, and / or other hardware (or their corresponding virtual computing resources) to perform training and / or inference by using the above resources. In addition, one or more of the software and / or hardware resources described above may be configured to allow a user to train or perform a service for inferring information, such as image recognition, speech recognition, or other artificial intelligence services.

[0217] Example network environment

[0218] A network environment suitable for implementing embodiments of the present disclosure may include one or more client devices, servers, network attached storage (NAS), other backend devices, and / or other device types. Client devices, servers, and / or other device types (e.g., each device) may be implemented on one or more instances of Figure 11 one or more computing devices 1100 - for example, each device may include one or more components, features, and / or functions similar to those of computing device 1100. In addition, in the case of implementing a backend device (e.g., a server, NAS, etc.), the backend device may be included as part of data center 1200, an example of which is described in more detail herein with respect to Figure 12 more details.

[0219] Components of the network environment may communicate with each other via a network, which may be wired, wireless, or both. The network may include one network among a plurality of networks or more networks. For example, the network may include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the public switched telephone network (PSTN)), and / or one or more private networks. In the case where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) may provide a wireless connection.

[0220] Compatible network environments can include one or more peer-to-peer network environments (in which case, a server may not be included in the network environment) and one or more client-server network environments (in which case, one or more servers may be included in the network environment). In a peer-to-peer network environment, the functions described herein for a server can be implemented on any number of client devices.

[0221] In at least one embodiment, the network environment can include one or more cloud-based network environments, distributed computing environments, combinations thereof, etc. A cloud-based network environment can include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer can include a framework that supports software layers and / or one or more applications of an application layer. The software or application can respectively include network-based service software or applications. In an embodiment, one or more client devices can use network-based service software or applications (e.g., by accessing service software and / or applications via one or more application programming interfaces (APIs)). The framework layer can be, but is not limited to, a free and open-source software web application framework that can perform large-scale data processing (e.g., "big data") using a distributed file system.

[0222] A cloud-based network environment can provide any combination of cloud computing and / or cloud storage that performs the computing and / or data storage functions (or one or more parts thereof) described herein. Any of these different functions can be distributed across multiple locations from a central or core server (e.g., one or more data centers that can be located in a state, region, country, globally, etc.). If the connection to a user (e.g., a client device) is relatively close to an edge server, the core server can assign at least a portion of the function to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0223] One or more client devices can include those described herein with respect to Figure 11At least some of the components, features, and functions of the one or more example computing devices 1100 described. By way of example, and not limitation, a client device may be implemented as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, camera, surveillance device or system, vehicle, boat, spacecraft, virtual machine, drone, robot, handheld communication device, hospital device, gaming device or system, entertainment system, vehicle computer system, embedded system controller, remote control, appliance, consumer electronic device, workstation, edge device, any combination of these depicted devices, or any other suitable device.

[0224] The present disclosure may be described in the general context of machine - usable instructions or computer code, including computer - executable instructions such as program modules, being executed by a computer or other machine such as a personal digital assistant or other handheld device. Generally, program modules, including routines, programs, objects, components, data structures, etc., refer to code that performs particular tasks or implements particular abstract data types. The present disclosure may be practiced in a variety of system configurations, including handheld devices, consumer electronics, general - purpose computers, more specialized computing devices, etc. The present disclosure may also be practiced in a distributed computing environment where tasks are performed by remote processing devices linked through a communication network.

[0225] As used herein, the recitation of "and / or" with respect to two or more elements should be interpreted to mean only one element or a combination of elements. For example, "element A, element B, and / or element C" may include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. Further, "at least one of element A or element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Still further, "at least one of element A and element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0226] This disclosure describes the subject matter of the present disclosure in detail to meet statutory requirements. However, the description itself is not intended to limit the scope of the present disclosure. On the contrary, the present disclosure has contemplated that the claimed subject matter may also be embodied in other ways, including steps different from those described herein or combinations of similar steps in combination with other current or future technologies. Moreover, although the terms "step" and / or "block" may be used herein to imply different elements of the methods employed, these terms should not be construed to imply any particular order among or between the various steps disclosed herein, unless the order of the steps is explicitly described.

[0227] Example paragraph

[0228] A. A method, comprising: determining one or more parameters associated with at least a portion of a simulation dataset; determining one or more values associated with the one or more parameters; using one or more simulation systems and generating at least a portion of the simulation dataset based at least on the one or more values; and generating data representing at least the one or more values associated with the one or more parameters and at least corresponding to a state or time associated with at least a portion of the simulation dataset based at least on generating at least a portion of the simulation dataset.

[0229] B. The method according to paragraph A, further comprising: determining one or more poses associated with one or more objects represented by at least a portion of the simulation dataset, wherein the data further represents one or more pose values of the one or more poses.

[0230] C. The method according to paragraph A or paragraph B, wherein: determining the one or more values associated with the one or more parameters comprises: sampling at least a first parameter among the one or more parameters to determine a first value among the one or more values, and then sampling a second parameter among the one or more parameters to determine a second value among the one or more values; and the data further represents an order that includes: the first value associated with the first parameter, followed by the second value associated with the second parameter.

[0231] D. The method according to any one of paragraphs A-C, further comprising: determining one or more assets associated with the one or more parameters, wherein the data further represents at least one of the one or more assets or one or more versions associated with the one or more assets.

[0232] E. A method according to any one of paragraphs A - D, wherein the one or more values include at least one of the following: one or more random values associated with one or more first parameters among the one or more parameters; or one or more set values associated with one or more second parameters among the one or more parameters.

[0233] F. A method according to any one of paragraphs A - E, wherein the at least part of the simulated data set is a first part of the simulated data set, and wherein the method further comprises: generating a second part of the simulated data set based at least on one or more second values associated with one or more second parameters, wherein the data also represents the one or more second values associated with the one or more second parameters.

[0234] G. A method according to any one of paragraphs A - F, further comprising: determining one or more random values generated by generating the at least part of the simulated data set, wherein the data also represents the one or more random values.

[0235] H. A method according to any one of paragraphs A - G, further comprising: regenerating the at least part of the simulated data set based at least on the one or more values associated with the one or more parameters represented by the data.

[0236] I. A method according to any one of paragraphs A - H, further comprising: generating updated data by modifying at least one value associated with at least one parameter among the one or more parameters represented by the data; and generating at least part of a second simulated data set related to the at least part of the simulated data set based at least on the updated data.

[0237] J. A method according to any one of paragraphs A - I, further comprising: generating parameter data representing at least one or more second parameters and one or more second values associated with the one or more second parameters; and generating at least part of a second simulated data set related to the at least part of the simulated data set based at least on the data and the parameter data.

[0238] K. A system comprising: one or more processing units, the one or more processing units configured to: determine at least a portion of a first simulation dataset for recreation; obtain data representing one or more values associated with one or more parameters used to generate the at least a portion of the first simulation dataset; and generate at least a portion of a second simulation dataset similar to the at least a portion of the first simulation dataset based at least on the one or more values associated with the one or more parameters.

[0239] L. The system according to paragraph K, wherein: the data represents an order that at least includes: a first parameter among the one or more parameters, followed by a second parameter among the one or more parameters; and generating the at least a portion of the second simulation dataset at least includes: sampling the first parameter based at least on the order represented by the data to determine a first value among the one or more values, and subsequently sampling the second parameter to determine a second value among the one or more values so as to generate the at least a portion of the second simulation dataset.

[0240] M. The system according to paragraph K or paragraph L, wherein: the data further represents one or more poses associated with one or more objects represented by the at least a portion of the first simulation dataset; and generating the at least a portion of the second simulation dataset further includes at least based on one or more pose values of the one or more poses.

[0241] N. The system according to any one of paragraphs K - M, wherein the one or more processing units are further configured to: generate updated data by modifying at least one value among the one or more values associated with at least one parameter among the one or more parameters represented by the data, wherein generating the at least a portion of the second simulation dataset is at least based on the updated data.

[0242] O. The system according to any one of paragraphs K - N, wherein the one or more processing units are further configured to: generate parameter data representing one or more second values associated with one or more second parameters, wherein generating the at least a portion of the second simulation dataset is further at least based on the one or more second values associated with the one or more second parameters represented by the parameter data.

[0243] P. The system according to any one of paragraphs K - O, wherein the data further represents one or more assets associated with the one or more parameters; and generating the at least a portion of the second simulation dataset is further at least based on the one or more assets represented by the data.

[0244] Q. The system according to any one of paragraphs K - P, wherein the one or more processing units are further configured to: determine the one or more parameters associated with the first simulated data set; determine the one or more values associated with the one or more parameters; generate the first simulated data set based at least on the one or more values; and generate data representing at least the one or more values associated with the one or more parameters based at least on generating the first simulated data set.

[0245] R. The system according to any one of paragraphs K - Q, wherein the system is included in at least one of the following: a control system for an autonomous or semi - autonomous machine; a perception system for an autonomous or semi - autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing optical transmission simulation; a system for performing collaborative content creation of 3D assets; a system for performing deep learning operations; a system implemented using edge devices; a system implemented using robots; a system implementing one or more language models; a system implementing one or more large language models (LLMs); a system for performing conversational AI operations; a system for generating synthetic data; a system for performing AI operations; a system including one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

[0246] S. A processor comprising: one or more processing units configured to generate log data associated with the generation of a simulated scenario, wherein the log data represents at least the one or more parameters sampled for generating the simulated scenario and the one or more values associated with the one or more parameters.

[0247] T. The processor according to paragraph S, wherein the processor is included in at least one of the following: a control system for an autonomous or semi - autonomous machine; a perception system for an autonomous or semi - autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing optical transmission simulation; a system for performing collaborative content creation of 3D assets; a system for performing deep learning operations; a system implemented using edge devices; a system implemented using robots; a system implementing one or more language models; a system implementing one or more large language models (LLMs); a system for performing conversational AI operations; a system for generating synthetic data; a system for performing AI operations; a system including one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

Claims

1. A method comprising: determining one or more parameters associated with at least a portion of the simulated data set; determining one or more values ​​associated with the one or more parameters; generating said at least a portion of said simulated data set using one or more simulation systems and based at least on said one or more values; as well as Based at least on generating the at least a portion of the simulated data set, data representing at least the one or more values ​​associated with the one or more parameters and corresponding to at least a state or time associated with the at least a portion of the simulated data set is generated.

2. The method according to claim 1, further comprising: determining one or more poses associated with one or more objects represented by the at least a portion of the simulation data set, The data further represents one or more gesture values ​​of the one or more gestures.

3. The method according to claim 1, wherein: Determining the one or more values ​​associated with the one or more parameters includes: sampling at least a first parameter of the one or more parameters to determine a first value of the one or more values, and then sampling a second parameter of the one or more parameters to determine a second value of the one or more values; and The data further represents a sequence comprising: the first value associated with the first parameter followed by the second value associated with the second parameter.

4. The method according to claim 1, further comprising: determining one or more assets associated with the one or more parameters, The data further represents at least one of the one or more assets or one or more versions associated with the one or more assets.

5. The method of claim 1 , wherein the one or more values ​​include at least one of: one or more random values ​​associated with one or more first parameters of the one or more parameters; or One or more set values ​​associated with one or more second parameters of the one or more parameters.

6. The method of claim 1 , wherein the at least a portion of the simulated data set is a first portion of the simulated data set, and wherein the method further comprises: generating a second portion of the simulated data set based at least on one or more second values ​​associated with one or more second parameters, Wherein the data further represents the one or more second values ​​associated with the one or more second parameters.

7. The method according to claim 1, further comprising: determining one or more random values ​​resulting from generating said at least a portion of said simulated data set, Wherein the data also represents the one or more random values.

8. The method according to claim 1, further comprising: The at least a portion of the simulated data set is regenerated based at least on the one or more values ​​associated with the one or more parameters represented by the data.

9. The method according to claim 1, further comprising: generating updated data by modifying at least one of the one or more values ​​associated with at least one of the one or more parameters represented by the data; as well as At least a portion of a second simulated data set is generated that is related to the at least a portion of the simulated data set based at least on the updated data.

10. The method according to claim 1, further comprising: generating parameter data representing at least one or more second parameters and one or more second values ​​associated with the one or more second parameters; as well as At least a portion of a second simulated data set is generated that is related to the at least a portion of the simulated data set based at least on the data and the parameter data.

11. A system comprising: One or more processing units, the one or more processing units being configured to: determining at least a portion of a first simulation data set for re-creation; obtaining data representing one or more values ​​associated with one or more parameters used to generate the at least a portion of the first simulation data set; as well as At least a portion of a second simulated data set is generated that is similar to the at least a portion of the first simulated data set based at least on the one or more values ​​associated with the one or more parameters.

12. The system of claim 11, wherein: The data represents a sequence comprising at least: a first parameter of the one or more parameters followed by a second parameter of the one or more parameters; and Generating at least a portion of the second simulated data set includes at least sampling the first parameter to determine a first value among the one or more values ​​based at least on the order represented by the data, and then sampling the second parameter to determine a second value among the one or more values, so as to generate at least a portion of the second simulated data set.

13. The system of claim 11, wherein: The data also represents one or more poses associated with one or more objects represented by the at least a portion of the first simulation data set; as well as Generating the at least a portion of the second simulation data set is also based at least on one or more gesture values ​​of the one or more gestures.

14. The system of claim 11, wherein the one or more processing units are further configured to: generating updated data by modifying at least one of the one or more values ​​associated with at least one of the one or more parameters represented by the data, Wherein generating said at least a portion of said second simulated data set is based at least on said updated data.

15. The system of claim 11, wherein the one or more processing units are further configured to: generating parameter data representing one or more second values ​​associated with one or more second parameters, Wherein generating the at least a portion of the second simulation data set is also based at least on the one or more second values ​​associated with the one or more second parameters represented by the parameter data.

16. The system of claim 11, wherein: The data also represents one or more assets associated with the one or more parameters; as well as Generating the at least a portion of the second simulated data set is also based at least on the one or more assets represented by the data.

17. The system of claim 11, wherein the one or more processing units are further configured to: determining the one or more parameters associated with the first simulation data set; determining the one or more values ​​associated with the one or more parameters; generating the first simulated data set based at least on the one or more values; as well as Based at least on generating the first simulation data set, data representing at least the one or more values ​​associated with the one or more parameters is generated.

18. The system of claim 11, wherein the system is included in at least one of the following: control systems for autonomous or semi-autonomous machines; Perception systems for autonomous or semi-autonomous machines; A system for performing simulation operations; Systems for performing digital twin operations; A system for performing light transport simulations; A system for performing collaborative content creation of 3D assets; Systems for performing deep learning operations; Systems implemented using edge devices; Systems implemented using robots; A system implementing one or more language models; A system implementing one or more large language models (LLMs); Systems for performing conversational AI operations; Systems for generating synthetic data; Systems for performing AI operations; A system comprising one or more virtual machines VM; A system implemented at least in part in a data center; or A system implemented at least in part using cloud computing resources.

19. A processor comprising: One or more processing units, the one or more processing units are used to generate log data associated with the generation of a simulation scenario, wherein the log data at least represents one or more parameters sampled for generating the simulation scenario and one or more values ​​associated with the one or more parameters.

20. The processor of claim 19, wherein the processor is included in at least one of the following: control systems for autonomous or semi-autonomous machines; Perception systems for autonomous or semi-autonomous machines; A system for performing simulation operations; Systems for performing digital twin operations; A system for performing light transport simulations; Systems for performing collaborative content creation of 3D assets; Systems for performing deep learning operations; Systems implemented using edge devices; Systems implemented using robots; A system implementing one or more language models; A system implementing one or more large language models (LLMs); a system for performing conversational AI operations; Systems for generating synthetic data; Systems for performing AI operations; A system comprising one or more virtual machines VM; A system implemented at least in part in a data center; or a system implemented at least in part using cloud computing resources.

Images