Unsupervised learning of scene structures for synthetic data generation

Abstract

The invention discloses unsupervised learning of scene structures for synthetic data generation. A rule set or scene grammar may be used to generate a scene graph representing structures and visual parameters of objects in the scene. The renderer may take the scene graph as input and have a content library of assets identified in the scene graph, and may generate a scene composite image having a desired scene structure without the need to manually place any object in the scene. The image or environment synthesized in this manner may be used, for example, to generate training data for real-world navigation applications, and to generate virtual worlds for gaming or virtual reality experiences.

Description

[0001] This application is a divisional application of Chinese patent application No. 202110245600.4 filed on March 5, 2021.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This patent application is a continuation application and claims priority to U.S. Patent Application No. 17 / 117,425 filed on December 10, 2020, which claims the benefit of U.S. Provisional Patent Application No. 62 / 986,614 filed on March 6, 2020, both of which are incorporated herein by reference in their entirety. Background Art

[0004] Applications such as gaming, animation, and simulation increasingly rely on more detailed and realistic virtual environments. In many cases, procedural models are used to synthesize scenes in these environments, as well as to create labeled synthetic datasets for machine learning. To produce realistic and diverse scenes, experts must carefully tune the many parameters that control the procedural models. These parameters control both the structure of the generated scenes (e.g., how many cars are in a scene) and the parameters that place objects into valid configurations. The complexity and amount of knowledge required to manually determine and tune these parameters, as well as other aspects of configuring these scenes, can limit widespread applicability and may also limit the realism or scope of the generated environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Various embodiments according to the present disclosure will be described with reference to the accompanying drawings, in which:

[0006] Figure 1A and Figure 1B illustrates an image that may be generated according to at least one embodiment;

[0007] Figure 2A , Figure 2B and Figure 2C Rules and graphics showing a scenario according to at least one embodiment;

[0008] Figure 3A , Figure 3B and Figure 3C illustrates the stages of scene graph generation according to at least one embodiment;

[0009] Figure 4 A process for generating an image from a scene grammar according to at least one embodiment is shown;

[0010] Figure 5 A process for training a network according to at least one embodiment is shown;

[0011] Figure 6Components of a system for generating scene graphs according to at least one embodiment are shown;

[0012] Fig. 7A Inference and / or training logic according to at least one embodiment is shown;

[0013] Figure 7B Inference and / or training logic according to at least one embodiment is shown;

[0014] Figure 8 An example data center system is shown in accordance with at least one embodiment;

[0015] Fig. 9 A computer system according to at least one embodiment is shown;

[0016] Fig.10 A computer system according to at least one embodiment is shown;

[0017] Fig.11 illustrates at least a portion of a graphics processor according to one or more embodiments;

[0018] Fig.12 illustrates at least a portion of a graphics processor according to one or more embodiments;

[0019] Fig.13 is an example data flow diagram for an advanced computing pipeline according to at least one embodiment;

[0020] Fig.14 is a system diagram of an example system for training, tuning, instantiating, and deploying machine learning models in an advanced computing pipeline according to at least one embodiment; and

[0021] Fig.15A and Fig. 15B A data flow diagram of a process for training a machine learning model, and a client-server architecture for enhancing an annotation tool with a pre-trained annotation model, according to at least one embodiment, is shown. DETAILED DESCRIPTION

[0022] Methods according to various embodiments may provide for the generation of synthetic images and datasets. In particular, various embodiments may generate virtual scenes or environments based at least in part on a set of rules that define the placement and appearance of objects or "assets" within that environment. These datasets may be used to generate virtual environments, as well as to generate large training datasets related to target real-world datasets. Synthetic datasets provide attractive opportunities for training machine learning models that can be used for tasks such as perception and planning in autonomous and semi-autonomous driving, indoor scene perception, generative content creation, and robotic control. Through graphics engines, synthetic datasets can provide ground truth data for tasks where labels are expensive or even unavailable (e.g., segmentation, depth, or material information). As Figure 1A and Figure 1B As shown in the images 100, 150 of FIG. 1 , this can include ground truth data for objects rendered in those images, such as bounding boxes and labels for cars 102, 152 and people 104 rendered in those images. Adding new types of labels to such synthetic datasets can be performed by invoking the renderer, rather than embarking on a time-consuming annotation effort that requires new tools and hiring, training, and supervising annotators.

[0023] Using conventional methods, creating synthetic datasets has various obstacles. While the content that makes up a scene (e.g., three-dimensional computer-aided design (3D CAD) models that make up the scene) can be obtained from sources such as online asset stores, artists must typically write complex procedural models to synthesize the scene by placing these assets in a realistic layout. This often requires browsing through large amounts of real-world imagery to carefully adjust the procedural model, which can be a very time-consuming task. For scenes such as street scenes, creating a synthetic scene associated with one city may require adjusting a procedural model made for another city from scratch. Methods according to various embodiments may attempt to provide automated methods for handling these and other such tasks.

[0024] In one approach, scene parameters in synthetically generated scenes can be optimized by exploiting the visual similarity of generated (e.g., rendered) synthetic data to real data. The scene structure and parameters can be represented by a scene graph, and the data is generated by sampling random scene structures (and parameters) from a given probabilistic grammar of the scene, and the scene parameters are modified using a learned model. Since this approach only learns scene parameters, there is still a simulation-to-reality gap in the scene structure. For example, one might find that the density of cars, people, and buildings in Manhattan is higher than in a quaint village in Italy. Other work on generative models for structured data (e.g., graphs and grammatical strings) requires large amounts of ground truth data for training to generate realistic samples. However, annotation of scene structure is very cumbersome and is therefore not available in most real datasets.

[0025] The method according to various embodiments may utilize a procedural generative model of a synthetic scene that is unsupervisedly learned from a real image. In at least one embodiment, one or more scene graphs may be generated object-by-object by learning to sample rule extensions from a given probabilistic scene grammar and generate scene parameters. Learning such tasks without supervision may be challenging, at least in part due to the discrete nature of the scene structure to be generated and the presence of indistinguishable renderers in the generation process. To this end, feature space divergence may be utilized to compare the generated (e.g., rendered) scene with the real scene, which may be determined for each scene. Such a method may allow for credit assignment trained by reinforcement learning. Experiments conducted on two synthetic and real datasets show that the method according to at least one embodiment significantly reduces the distribution gap between the scene structure in the generated data and the target data, improving the human prior on the scene structure by learning to closely align with the target structure distribution. On the real dataset, starting from the smallest human prior, the structural distribution in the real target scene can be almost accurately restored, which is noteworthy because the model can be trained without any labels. For example, we have demonstrated that object detectors trained on this generated data outperform detectors trained on data generated with human priors, and demonstrated improvements in distributional similarity measures for rendered images generated using real data.

[0026] Instead of inferring each scene as in previous methods, methods according to various embodiments can generate new data that resembles the target distribution. One approach is to learn to optimize a non-differentiable simulator using a variable upper bound on a GAN-like objective, or to optimize simulator parameters for a control task by directly comparing real and simulated trajectories. A method according to at least one embodiment can learn to generate discrete scene structures constrained by grammar while optimizing a distribution matching objective (using reinforcement learning) rather than using adversarial training. This approach can be used to generate large and complex scenes, as opposed to images of single objects or faces.

[0027] In at least one embodiment, a generative model consisting of graphs and trees can produce graphs with richer structures and greater flexibility than grammar-based models, but may not be able to produce grammatically correct graphs, such as program and scene graphs, for situations where the grammar is defined. Grammar-based methods have been used for various tasks, such as program translation, conditional program generation, grammar induction, and generative modeling of structures (e.g., molecules) with grammars. However, these methods assume access to ground truth graph structures for learning. Methods according to various embodiments can train models in an unsupervised manner without any ground truth scene graph annotations.

[0028] In at least one embodiment, a rule set may be generated or obtained for a virtual scene or environment to be generated. For example, an artist may generate or provide a rule set for a scene, or may select from a library of rule sets for various scenes. This may include, for example, browsing through a graphical interface for scene type options and selecting a scene type with an associated rule set, as the scene type may correspond to scene types such as European cities, American rural areas, dungeons, and the like. For a given virtual or "synthetic" scene, an artist may also create or obtain content for various objects (e.g., "assets") in the scene, such as models, images, textures, and other features that may be used to render those objects. The provided rules may indicate how these objects should relate to each other in a given scene.

[0029] For example, Figure 2A An example rule set 200 that can be utilized in accordance with various embodiments is shown. The rule set can include any number of rules, or in some embodiments can reach a maximum number. Each rule can define a relationship between at least two types of objects to be represented in the synthetic scene. This rule set applies to locations where there will be roads and sidewalks. As shown in rule set 200, the road can have lanes according to the first rule. According to other rules, there can be a single lane or multiple lanes of land, and each lane can be associated with a sidewalk and one or more cars. As shown in the figure, the rules can also define whether an object type can have one or more instances of that object type associated with a given object type. Another pair of rules indicates that there can be one or more people on the sidewalk in this scene.

[0030] Such rules can be used to generate one or more scene structures representing the scene to be generated. Figure 2B and Figure 2C Two example scene structures 230, 260 are shown in FIG. In each of these structures, roads are shown as primary parent nodes in a hierarchical tree structure. The rules in the rule set 200 and the relationships defined therein determine potential parent-child relationships that can be used to generate different tree structures that conform to those relationships. In at least one embodiment, the generative model can generate these scene structures from the rule set using an appropriate sampling or selection process. Figure 2B In structure 230, there is a two-lane road, each lane has a sidewalk, and there is a car in one of the lanes. There is also a tree and a person on the sidewalk near the lane with the car. Figure 2CIn the structure 260 of , there is a single lane road with three cars and a sidewalk, with two people and a tree near the sidewalk. It can be seen that these structures represent two different scenes generated from the same rule set. This method can be used to build, supplement, expand, generate or synthesize a virtual environment that includes variations of object structures that all follow the selected rule set. Using this method, a single rule set can be used to generate an environment as large as required, and its changes can be random or can follow a variation strategy without the user manually selecting or placing these objects. This method can be used to generate synthetic scenes from real images in an unsupervised manner. In at least one embodiment, this method can learn a generative model of scene structure, from which samples (with other scene parameters) can be rendered to create synthetic images and labels. In at least one embodiment, this method can be used to generate synthetic training data with appropriate labels using unlabeled real data. A rule set and unlabeled real data can be provided as input to a generative model, and the generative model can generate different sets of scene structures.

[0031] The method according to at least one embodiment can learn such a generative model for synthesizing scenes. In particular, given a real image X R The problem is to create a dataset representing X R Synthetic data D(θ)=(X(θ),Y(θ)) of image X(θ) and label Y(θ), where θ represents the parameters of the generative model. In at least one embodiment, by stipulating that the synthetic data D is the output of creating an abstract scene representation and rendering the scene representation with a graphics engine, advances in graphics engines and rendering can be utilized. Rendering can ensure that low-level pixel information in X(θ) (and its corresponding annotation Y(θ)) does not need to be modeled. Ensuring the semantic validity of the sampled scene may require imposing at least some constraints on its structure. Scene grammars use rule sets to greatly reduce the space of scenes that can be sampled, making learning a more structured and tractable problem. For example, scene grammars can explicitly enforce that cars can only be on the road, and then there is no need for implicit learning. Methods according to various embodiments can partially exploit this by using probabilistic scene grammars. The scene graph structure can be sampled from a prior imposed on a probabilistic context-free grammar (PCFG), which is referred to as a structural prior in this article. Parameters can be sampled for each node in the scene graph from the parameter prior and learn to predict new parameters for each node, thereby keeping the structure intact. Therefore, the resulting generated scenes come from structural priors (which are context-free) and learned parameter distributions, which may lead to simulation-to-real gaps in the scene structure.

[0032] Methods according to various embodiments can at least alleviate this gap by learning an unsupervised context-dependent structure distribution of synthetic scenes from images. In at least one embodiment, one or more scene graphs can be used as abstract scene representations that can be rendered into corresponding images with labels. FIG. 3A to FIG. 3C Components that may be used at different stages of such a process are shown. Figure 3A A set of logits 300 generated from a regular sample is shown, where a given sample is used to determine the next logit. Figure 3B The corresponding mask 330 that can be used to generate the scene graph is shown. In the process of scene graph generation, the shape of logit and mask is T max ×K. In Figure 3, unpatterned (e.g., pure white) areas represent higher values, while patterned areas represent lower values. Such a process can automatically regressively sample rules at each time step and predict the logit of the next rule conditioned on the sample, thereby capturing contextual dependencies. Sampling can be used to generate scene structure 362, such as Figure 3C As shown, and the parameters of the nodes that determine the scene structure. These parameters may include information such as position, height and posture. These and other parameters 366 can be sampled and applied to each node in the scene structure to generate a complete scene graph. Therefore, such a process can utilize the sampling rules in the grammar and convert it into a graph structure. In this example, only objects that can be rendered are separated from the complete grammar string. The parameters of each node can be sampled from the prior, or learning can be selected. The generated scene graph can be rendered as shown. Such a generation model can sequentially sample extension rules from a given probabilistic scene grammar to generate a rendered scene graph. The model can be trained unsupervised and by reinforcement learning using a distribution divergence based on feature matching specifically designed to adapt to this setting.

[0033] Scene graphs may be advantageous in at least some embodiments due to their ability to describe scenes in a concise hierarchical manner in fields such as computer graphics and vision, where each node describes an object and its parameters in the scene. Parameters may relate to aspects such as 3D assets or poses. Parent-child relationships may define the parameters of child nodes relative to their parent nodes, enabling simple scene editing and manipulation. Additionally, cameras, lighting, weather, and other effects may be encoded into the scene graph. Generating the corresponding pixels and annotations may be equivalent to placing the object in a scene in a graphics engine and rendering it with the defined parameters.

[0034] In at least one embodiment, a rule set may be defined as a vector having a length equal to the number of rules. A network may then be used to determine which of these rules to expand, and these rules may be expanded sequentially at different time steps. For each scene structure to be generated, a category distribution may be generated over all relevant rules in the set. The generative network may then sample from this categorical distribution to select rules for the scene, in which the categorical distribution may also be masked so that certain rules are forced to have zero probability, thereby preventing them from being selected. The network may also infer rules or options to be expanded for each object. In at least one embodiment, the generative model may be a recursive neural network (RNN). The potential vector defining the scene may be input to this RNN based on a determined probability to sequentially generate and expand the rules for the scene. The RNN moves down the tree or stack until all rules are processed (or the maximum number of rules is reached).

[0035] In one or more embodiments, Figure 3A Each row in can correspond to a sampling rule. Figure 3B As shown, the mask 330 can then be used to indicate to the model which rules will be extended at a given time step. This process can be performed iteratively to generate a valid scene description. In addition, the relationships between objects in the scene graph also provide geometric constraints for the scene, because these objects cannot exist outside of the specified relationships, such as a car cannot be placed outside of a lane or on a sidewalk. The parameters of the nodes define various visual properties, so that a road in a rural New Zealand village looks different from a road in a large city in Thailand. In at least some embodiments, ranges can be set for these various parameters for specific types of objects, so that sidewalks are only accessible at a certain width, roads have only a limited number of lanes, and so on.

[0036] These data structures can also be used to perform additional learning. For example, this data can be used downstream to train a model to, for example, detect cars in captured image data. For example, this structure can be preserved with generated images to help the model more quickly identify cars based on where they appear in the scene structure.

[0037] In at least one embodiment, a context-free grammar G can be defined as a list of symbols (e.g., terminals and non-terminals) and expansion rules. Non-terminal symbols have at least one expansion rule that can be expanded into a new set of symbols. Sampling from the grammar may involve expanding a start symbol (or initial or parent symbol) until only non-terminal symbols remain. A total number of expansion rules K can be defined in the grammar G. A scene grammar can be defined and strings can be sampled from the represented grammar using one or more scene graphs. For each scene graph, a structure T can be sampled from the grammar G and then corresponding parameters α can be sampled for each node in the graph. In at least one embodiment, a convolutional network is used with the scene graph to sample a set of parameters for each individual node in the graph.

[0038] In various approaches, generative models with grammatically constrained graphs can be utilized. In at least one embodiment, a recurrent neural network can be used to map the latent vector z to unnormalized probabilities over all possible grammatical rules in an autoregressive manner. In such an embodiment, this can persist for a maximum of T max In at least one embodiment, a regular r can be sampled at each time step. t , and this rule can be used to predict the next rule f t+1 This allows the model to easily capture context-dependent relationships, compared to the context-independent nature of traditional methods for scene graphs. Given at most T max A list of sampling rules is constructed and the corresponding scene graph is generated by treating each rule expansion as a node expansion in this graph, as shown in Figure 3.

[0039] To ensure the validity of these sampling rules at each time step t, a last-in-first-out (LIFO) stack of unexpanded non-terminal nodes can be maintained. A node can be popped from the stack and expanded according to the sampled rule expansion, and the resulting new non-terminal node can be pushed onto the stack. When popping a non-terminal, a mask m of size K can be created t , for a valid rule from this non-terminal, the mask m t is 1, otherwise 0. Given the next expansion f t Logit, rule r t,k The probability of can be given by:

[0040]

[0041] Sampling from this masked multinomial distribution ensures that only valid rules are sampled as r t Given logit and sampling rule The probability of the corresponding scene structure T given z can be given by:

[0042]

[0043] In summary, we can sample the scene structure T~q from the model θ (·|z), then sample the parameters of each node in the scene α~q(·|T) and render the image v'=R(T,α)~q I , to generate an image. For some v'~q with parameter α and structure T I , we can make the following assumptions:

[0044] q I (v′|z)=q(α|T)q θ (T|z)

[0045] Various training methods can be used for such a generative model. In at least one embodiment, the training can be performed using variational inference or by optimizing a measure of distributional similarity. Variational inference allows the use of reconstruction-based objectives by introducing an approximate learning posteriori. Training such a model using variational inference can be challenging, at least in part due to the complexity from discrete sampling and having a renderer in the generation process. In addition, the recognition network here may be equivalent to doing inverse graphics, which is itself a very challenging problem. In at least one embodiment, a measure of distributional similarity of the generated data and the target data can be optimized. Adversarial training of generative models can be used in conjunction with reinforcement learning (RL), for example by carefully limiting the capabilities of the critic. In at least one embodiment, reinforcement learning can be used to train discrete generative models in scene graphs. Metrics can be calculated for each sample, which can significantly improve the overall training process.

[0046] The generative model can be trained to match the feature distribution of real data in the latent space of some feature extractor φ. For some v~p I , can be To define the true feature distribution. Similarly, for some v~q I , the generated feature distribution can be defined as In at least one embodiment, p can be calculated by approximating it from the sample f ,q f And minimize from p f to q f The KL divergence of can achieve distribution matching. In at least one embodiment, the training objective can be given in the following way:

[0047]

[0048] Using the above feature distribution definition, an equivalent objective can be given by:

[0049]

[0050] The true underlying feature distribution q f and p f In at least one embodiment, an approximation calculated using kernel density estimation (KDE) can be used and An example method can make V = {v1, ..., v l} and B={v'1,...,v' m} is a batch of real and generated images. KDE is performed with B and V to estimate q f , p f produce:

[0051]

[0052] Where K H is the standard multivariate normal kernel of the bandwidth matrix H. Here, H = dI can be used, where d is the dimension of the feature space.

[0053] A generative model according to at least one embodiment may make discrete (e.g., non-differentiable) choices at each step, such that it may be advantageous to use reinforcement learning techniques to optimize the objective. Specifically, this may include using a reinforcement score function estimator with a moving average baseline, whereby a gradient may be given by:

[0054]

[0055] Where M is the batch size, and is the density estimate defined above.

[0056] It can be noted that the above gradient needs to calculate the edge probability q of the generated image v' I (v'), instead of condition q I (v'|z). Computing the marginal probability of the generated image involves intractable marginalization on the latent variable z. To avoid this, a fixed finite number of latent vectors from the set Z can be used, which are uniformly sampled and thus easy to marginalize. This is converted to:

[0057]

[0058] q I (v′)=q(α|T)q θ (T|Z)

[0059] Such an approach can still provide sufficient modeling capability because only the maximum length T maxThere are finitely many scene graphs that can be sampled from the grammar. As a rule of thumb, using a single latent vector is sufficient, since the randomness in the regular sampling makes up for the loss of randomness in the latent space.

[0060] In at least one embodiment, pre-training can be an important step. Hand-crafted priors can be defined on the scene structure. For example, a simple prior can be to place a car on a road in a driving scene. The model can be pre-trained at least in part by sampling strings (e.g., scene graphs) from grammatical priors, and training the model to maximize the log-likelihood of these scene graphs. Feature extraction can also be an important step in distribution matching, as features need to capture structural scene information, such as the number of objects and their contextual spatial relationships, for effective training.

[0061] During model training, sampling may result in using a maximum of T max The string generated by the step size is incomplete. Therefore, the scene graph T can be repeatedly sampled until its length is the maximum T max To ensure that this does not require too many attempts, when sampling a single scene graph used to generate F, the rejection rate r of the sampled features F can be reject (F) is recorded as the average number of failed sampling attempts. reject (F) Setting the threshold f to represent the maximum allowed rejection and the weight λ, which can then be added to the initial loss, can be given by the following formula:

[0062]

[0063] According to experience, it is found that in at least one embodiment, λ=10 -2 And values ​​of f=1 work well.

[0064] This method can provide unsupervised learning of a generative model of synthetic scene structure by optimizing visual similarity to real data. Even if annotations are provided, it is difficult to infer scene structure. Methods according to various embodiments can perform this generation part without any ground truth information. Experiments have verified the ability of such a model to learn a reasonable prior on the scene structure, which is a significant improvement over the manually designed prior. In order to produce satisfactory results, the method can optimize the scene structure and parameters of the synthetic scene generator.

[0065] As mentioned, this method for generating various scene graphs can enable the generation of scenes or environments that mimic the real world or target world or environment. Information about this world or environment can be learned directly from the pixels of the example images of the real world or target world. Such methods can be used to attempt accurate reconstruction, but in many embodiments, it can allow the generation of an infinite variety of worlds and environments that can be based at least in part on these real worlds or target worlds. A set of rules or scene grammars can be provided to describe the world at a microscopic level, defining object-specific relationships. For example, the person can specify or select rules that can be used to automatically generate the scene or image without having to manually generate at least a layout for each scene or image. The scene graph in at least one embodiment can provide a complete description of the layout of the three-dimensional world. In at least one embodiment, the recursive expansion of the rules for generating the scene structure can also be used to generate a string that provides a complete representation or definition of the layout of the three-dimensional scene. As previously described, a generative model can be used to perform the expansion and generate the scene structure. In at least one embodiment, the scene structure can be stored as a JSON file or using another such format. This JSON file can then be provided as input to a rendering engine to generate an image or scene. The rendering engine can extract appropriate asset data for rendering individual objects.

[0066] As mentioned, the rendered data can be used to render a virtual environment, such as for a game or VR application. The rendering can also be used to generate training data for such applications and other applications (e.g., training models for autonomous or semi-autonomous machines, such as training models for vehicle navigation or robotics simulation). Different asset libraries can be selected for these renderings so that the environment may be suitable for different geographic locations, time points, etc. Each pixel in the rendered image can be marked to indicate the type of object represented by the pixel. In at least one embodiment, a bounding box or other position indicator can be generated for each object, as well as a depth determined for each pixel in the 3D scene, a normal at the pixel location, etc. In at least some embodiments, this information can be extracted from the rendering engine by extracting the data using an appropriate rendering function.

[0067] In at least one embodiment, an artist can provide an asset set and select a rule set, and the entire virtual environment can be generated without manual input from the artist. In some embodiments, there may be a library of assets that the artist can choose from. For example, an artist can select a scene structure for a "Japanese city" and select assets for a "Japanese city" in order to generate an environment based on a Japanese city, including appropriate visual objects and layouts, but not directly corresponding to or representing any Japanese city. In some embodiments, the artist may have the ability to adjust the environment by indicating what the artist likes or dislikes. For example, for this application, the artist may not want cars on the street. Therefore, the artist can indicate that the artist does not want cars to be included or at least not included in a specific area or associated with a specific object type, and a new scene graph can be generated that has removed the car and updated the appropriate relationship. In some embodiments, a user can provide, obtain, utilize or generate two or more sub-graphs, such as things that can indicate what the user likes and what the user does not like. These sub-graphs can then be used to generate new scenes that better meet the user's expectations. This method can make it easy for users to generate virtual environments with specific aspects and visual appearances without any professional knowledge of the creation process, and without manually placing, moving or adjusting objects in the scene or scene set. Such a method can enable ordinary people to become 3D artists with minimal effort.

[0068] Figure 4An example process 400 for generating an image of a scene that can be utilized is shown according to various embodiments. It should be understood that for this process and other processes presented herein, unless otherwise specifically stated, there may be additional, fewer or alternative steps that are performed in a similar or alternative order or at least partially in parallel within the scope of various embodiments. In this example, a set of rules to be used to generate at least one scene is determined 402. This can include a user generating these rules or selecting from a number of rule sets, as well as other such options. The individual rules in the set can define the relationship between object types in the scene. The rule set can be sampled 404 based on a determined probability to generate a scene structure including the relationship of the objects defined by the rules. In at least one embodiment, this can include a hierarchical scene structure, where the nodes of the hierarchical structure correspond to the object types of the scene. Parameters for rendering each of these objects can be determined 406, such as by sampling from an appropriate data set. A scene graph can then be generated 408, which can be based on the scene structure, but with appropriate parameters applicable to the individual nodes or objects. The scene graph can be provided 410 to a renderer 410 or other object together with an asset library or other source of object content to generate an image or scene. A rendered image of the scene rendered based on the determined scene graph may be received 412. If the image is used as training data as described herein, the rendered image may include object labels, as well as preserve the scene structure.

[0069] Various methods can be used to train the neural networks discussed in this article. For example, a generative model can be trained to analyze unlabeled images that may correspond to captured images of real-world settings. The generative network can then be trained to generate scenes with similar appearance, layout, and other such aspects. There can be both real scenes and synthetic scenes. Whenever these scenes are passed to a deep neural network, a set of features corresponding to locations in a high-dimensional space (e.g., a 1,000-dimensional space) can be extracted from the scene. The scene can then be considered to consist of these points in the high-dimensional space rather than pixels in the image. The network can be trained so that the features corresponding to the synthetic scene in the feature space are aligned with the features corresponding to the real scene. In this way, it may be difficult to distinguish the features of the real scene and the synthetic scene in the feature space.

[0070] In at least one embodiment, this can be achieved using reinforcement learning. As previously described, the goal can be to align two complete data sets, but there is no data or correlation with specific feature points that should be aligned, because this is an unsupervised space without correlation. Since the goal in many cases is not to generate an exact copy of the scene but to generate a similar scene, it is sufficient to align the distribution of feature points in this feature space. Therefore, the training process can compare real scenes and synthetic scenes as a whole. In order to evaluate the scene, the scene in the feature space can be compared with the distribution of feature points of other scenes. In various methods, it is difficult to determine whether the scene is realistic or useful by comparing the signal alone, except whether the structure is suitable. Therefore, the training method according to at least one embodiment can extract signals from each single data point itself without having to look at the entire data set. In this way, the signal can be evaluated for the degree to which a particular scene is aligned with the entire data set. Then the possibility of a particular scene resolving to all synthetic scenes can be calculated. This can provide the possibility that the particular scene is synthetic. This can be performed in at least one embodiment by using kernel density estimation (KDE). KDE can be used to obtain the probability that the scene belongs to the distribution of synthetic scenes. KDE can also be used to calculate the probability that the scene belongs to the distribution of real scenes. In at least one embodiment, the ratio of these values ​​can be analyzed and used to optimize the system. Maximizing this ratio (as a logarithm) as a reward function for the scenario can provide a signal that can be optimized for each individual scenario.

[0071] Figure 5 An example process 500 for training a network to generate realistic images that can be utilized is shown in accordance with at least one embodiment. In this example, scene graphs and assets such as those described above with respect to Figure 4 As described. The scene graph and the asset can be used to generate 504 a synthetic image of the scene. The location of the feature point in an n-dimensional feature space can be determined 506 for the generated image, where n can be equal to a number of rules in the set used to generate the scene graph. The feature point of the generated image can be compared 508 with the distribution of the feature points of the synthetic image in the feature space. A first probability can be determined 510 based on the comparison that the generated image is synthetic. The feature points of the generated image can also be compared 512 with the distribution of the feature points of the real image in the feature space. A second probability can be determined 514 based on the comparison that the generated image is real. The ratio of the two probabilities can be calculated 516, and one or more weights for the trained network can be adjusted to optimize the ratio.

[0072] Another embodiment may utilize the discriminator of the GAN. The GAN may be partially trained with the discriminator to determine whether the generated scene is real. The network may then be optimized so that the discriminator determines with high probability that the scene is real. However, this approach may be challenging because current renderers generate high-quality images, but these images may still be identified as not real captured images, so that the discriminator may mainly distinguish the difference even if the images are very similar in structure. In this case, the GAN may collapse during training because the discriminator cannot provide any valuable information because the rendered image will never confuse the discriminator as a real image. In at least one embodiment, an image-to-image conversion may be performed before providing the image data to the GAN to try to improve the appearance of these synthetic images. Image-to-image conversion can help narrow the style gap between real and synthetic images, helping to resolve low-level visual aspects that may be related to textures or reflections that may cause the image to appear to be a synthetic image rather than a real image. This may be advantageous for systems that utilize ray tracing, for example, to produce reflections and other lighting effects.

[0073] In another embodiment, a process can be used to ensure termination. A neural network can be defined to run for a certain number of steps, such as 150 steps for computational reasons. This number may be too low to fully generate a scene graph based on the large number of rules to be analyzed and expanded. Therefore, the generated scene graph will be incomplete and will result in inaccurate rendering. In at least one embodiment, the network can be allowed to run to its limit. If the limit is insufficient for a certain scene, the remaining features can be determined and a negative reward applied to the model to generate the feature again. Such an approach can result in a scene that does not include all the features originally expected, but ensures that a scene that matches the rendering limit can be rendered.

[0074] In at least one embodiment, the client device 602 can generate content for the session using components of a content application 604 on the client device 602 and data stored locally on the client device. In at least one embodiment, a content application 624 (e.g., an image generation or editing application) executing on a content server 620 can initiate a session associated with at least the client device 602, as can utilize a session manager and user data stored in a user database 634, and can cause a content manager 626 to determine content 632 and render it using a rendering engine (if required for this type of content or platform), and send it to the client device 602 using an appropriate transport manager 622 for transmission via download, streaming, or other such transmission channel. In at least one embodiment, this content 632 can include assets that can be used by the rendering engine to render a scene based on a determined scene graph. In at least one embodiment, the client device 602 receiving the content may provide the content to a corresponding content application 604, which may also or alternatively include a rendering engine for rendering at least some of the content for presentation via the client device 602, such as images or video content through a display 606 and audio (e.g., sounds and music) through at least one audio playback device 608 (e.g., speakers or headphones). In at least one embodiment, at least some of the content may already be stored on the client device 602, rendered on the client device 602, or accessible to the client device 602, so that at least that portion of the content does not need to be transmitted over the network 640, for example, the content may have been pre-downloaded or locally stored on a hard drive or optical disk. In at least one embodiment, a transmission mechanism such as data streaming may be used to transmit the content from the server 620 or content database 634 to the client device 602. In at least one embodiment, at least a portion of the content may be obtained or streamed from another source (e.g., a third-party content service 660), which may also include a content application 662 for generating or providing content. In at least one embodiment, multiple computing devices or multiple processors within one or more computing devices may be used to perform portions of the functionality, which may include a combination of a CPU and a GPU, for example.

[0075] In at least one embodiment, the content application 624 includes a content manager 626 that can determine or analyze the content before transmitting the content to the client device 602. In at least one embodiment, the content manager 626 can also include or work with other components capable of generating, modifying, or enhancing the content to be provided. In at least one embodiment, this can include a rendering engine for rendering image or video content. In at least one embodiment, a scene graph generation component 628 can be used to generate a scene graph from a set of rules and other such data. In at least one embodiment, an image generation component 630 that also includes a neural network can generate an image from the scene graph. In at least one embodiment, the content manager 626 can then cause the generated image to be transmitted to the client device 602. In at least one embodiment, the content application 604 on the client device 602 can also include components such as a rendering engine, a scene graph generator 612, and an image generation module 614 so that any or all of this functionality can be performed additionally or alternatively on the client device 602. In at least one embodiment, the content application 662 on the third-party content service system 660 can also include such functionality. In at least one embodiment, the location where at least some of the functions are performed may be configurable or may depend on factors such as the type of client device 602 or the availability of a network connection with appropriate bandwidth, among other such factors. In at least one embodiment, the system for content generation may include any suitable combination of hardware and software in one or more locations. In at least one embodiment, the generated image or video content at one or more resolutions may also be provided to or made available to other client devices 650, such as for downloading or streaming from a media source storing a copy of the image or video content. In at least one embodiment, this may include transmitting images of game content for a multiplayer game, where different client devices may display the content at different resolutions, including one or more super-resolutions.

[0076] In this example, the client devices may include any appropriate computing device, such as a desktop computer, a laptop computer, a set-top box, a streaming device, a game console, a smartphone, a tablet computer, a VR headset, an AR goggles, a wearable computer, or a smart TV. Each client device may submit a request on at least one wired or wireless network, which may include the Internet, an Ethernet, a local area network (LAN), or a cellular network, and other such options. In this example, the requests may be submitted to an address associated with a cloud provider, which may operate or control one or more electronic resources in a cloud provider environment, such as a data center or a server farm. In at least one embodiment, the request may be received or processed by at least one edge server located at the edge of the network and outside of at least one security layer associated with the cloud provider environment. In this way, latency may be reduced by enabling the client device to interact with a closer server, while also improving the security of resources in the cloud provider environment.

[0077] In at least one embodiment, such a system may be used to perform graphics rendering operations. In other embodiments, such a system may be used for other purposes, such as for performing simulation operations to test or verify autonomous machine applications, or for performing deep learning operations. In at least one embodiment, such a system may be implemented using an edge device, or may be incorporated into one or more virtual machines (VMs). In at least one embodiment, such a system may be implemented at least in part in a data center or at least in part using cloud computing resources.

[0078] Inference and training logic

[0079] Fig. 7A Inference and / or training logic 715 is shown for performing reasoning and / or training operations associated with one or more embodiments. Fig. 7A and / or Figure 7B Details regarding the inference and / or training logic 715 are provided.

[0080] In at least one embodiment, the inference and / or training logic 715 may include, but is not limited to, code and / or data storage 701 to store forward and / or output weights and / or input / output data and / or other parameters to configure neurons or layers of a neural network for training and / or for inference in aspects of one or more embodiments. In at least one embodiment, the training logic 715 may include or be coupled to code and / or data storage 701 to store graphics code or other software to control the timing and / or sequence in which weights and / or other parameter information will be loaded to configure logic including integer and / or floating point units (collectively referred to as arithmetic logic units (ALUs)). In at least one embodiment, the code (e.g., graphics code) loads weights or other parameter information into a processor ALU based on the architecture of the neural network to which the code corresponds. In at least one embodiment, the data storage 701 stores weight parameters and / or input / output data for each layer of a neural network trained or used in conjunction with one or more embodiments during forward propagation of input / output data and / or weight parameters during training and / or inference using aspects of one or more embodiments. In at least one embodiment, any portion of code and / or data storage 701 may be included within other on-chip or off-chip data storage, including a processor's L1, L2, or L3 cache or system memory.

[0081] In at least one embodiment, any portion of code and / or data storage 701 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, code and / or code and / or data storage 701 may be cache memory, dynamic random addressable memory ("DRAM"), static random addressable memory ("SRAM"), non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the choice of whether code and / or code and / or data storage 701 is internal or external to a processor, for example, or consists of DRAM, SRAM, flash memory, or some other storage type, may depend on the available storage space on-chip or off-chip, the latency requirements of the training and / or inference functions being performed, the batch size of data used in inference and / or training of a neural network, or some combination of these factors.

[0082] In at least one embodiment, the inference and / or training logic 715 may include, but is not limited to, code and / or data storage 705 to store backpropagation and / or output weights and / or input / output data corresponding to neurons or layers of a neural network trained and / or used for inference in aspects of one or more embodiments. In at least one embodiment, during training and / or inference using aspects of one or more embodiments, the code and / or data storage 705 stores weight parameters and / or input / output data for each layer of a neural network trained or used in conjunction with one or more embodiments during backpropagation of input / output data and / or weight parameters. In at least one embodiment, the training logic 715 may include or be coupled to the code and / or data storage 705 to store graphics code or other software to control the timing and / or sequence in which weights and / or other parameter information will be loaded to configure logic including integer and / or floating point units (collectively referred to as arithmetic logic units (ALUs)). In at least one embodiment, the code (e.g., graphics code) loads weights or other parameter information into a processor ALU based on the architecture of the neural network to which the code corresponds. In at least one embodiment, any portion of the code and / or data storage 705 may be included with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory. In at least one embodiment, any portion of the code and / or data storage 705 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the data storage 705 may be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the choice of whether the code and / or data storage 705 is internal or external to the processor, for example, consisting of DRAM, SRAM, flash memory, or some other storage type, depends on whether the available storage is on-chip or off-chip, the latency requirements of the training and / or inference functions being performed, the data batch size used in the inference and / or training of the neural network, or some combination of these factors.

[0083] In at least one embodiment, code and / or data storage 701 and code and / or data storage 705 may be separate storage structures. In at least one embodiment, code and / or data storage 701 and code and / or data storage 705 may be the same storage structure. In at least one embodiment, code and / or data storage 701 and code and / or data storage 705 may be partially the same storage structure and partially separate storage structures. In at least one embodiment, any portion of code and / or data storage 701 and code and / or data storage 705 may be included with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory.

[0084] In at least one embodiment, the inference and / or training logic 1015 may include, but is not limited to, one or more arithmetic logic units (“ALUs”) 1010 , including integer and / or floating point units, for performing logic and / or mathematical operations based at least in part on or as directed by training and / or inference code (e.g., graph code), which may produce activations (e.g., output values ​​from a layer or neuron within a neural network) stored in activation storage 720 , which is a function of input / output and / or weight parameter data stored in code and / or data storage 701 and / or code and / or data storage 705 . In at least one embodiment, activations are performed in response to executing instructions or other code, linear algebra and / or matrix-based mathematics performed by ALU 710 to generate activations stored in activation storage 720, where weight values ​​stored in code and / or data storage 705 and / or in code and / or data storage 701 are used as operands with other values, such as bias values, gradient information, momentum values, or other parameters or hyperparameters, any or all of which may be stored in code and / or data storage 705 or code and / or data storage 701 or other on-chip or off-chip storage.

[0085] In at least one embodiment, one or more ALUs 710 are included in one or more processors or other hardware logic devices or circuits, while in another embodiment, one or more ALUs 710 may be outside a processor or other hardware logic device or circuit that uses them (e.g., a coprocessor). In at least one embodiment, one or more ALUs 710 may be included within an execution unit of a processor, or otherwise included in a group of ALUs accessible by an execution unit of a processor, which may be within the same processor or distributed between different processors of different types (e.g., a central processing unit, a graphics processing unit, a fixed function unit, etc.). In at least one embodiment, code and / or data storage 701, code and / or data storage 705, and activation storage 720 may be on the same processor or other hardware logic device or circuit, while in another embodiment, they may be in different processors or other hardware logic devices or circuits or some combination of the same and different processors or other hardware logic devices or circuits. In at least one embodiment, any portion of activation storage 720 may be included with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory. Additionally, inference and / or training code may be stored with other code accessible to a processor or other hardware logic or circuitry and may be retrieved and / or processed using the processor's fetch, decode, schedule, execute, exit, and / or other logic circuitry.

[0086] In at least one embodiment, activation storage 720 may be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, activation storage 720 may be completely or partially internal or external to one or more processors or other logic circuits. In at least one embodiment, activation storage 720 may be selected to be internal or external to a processor, for example, or include DRAM, SRAM, flash memory, or other storage types, depending on the storage available on-chip or off-chip, the latency requirements for performing training and / or inference functions, the batch size of data used in inferencing and / or training neural networks, or some combination of these factors. In at least one embodiment, Fig. 7A The inference and / or training logic 715 shown in FIG. 7 may be used in conjunction with an application specific integrated circuit (“ASIC”), such as the ASIC from Google. Processing unit from Graphcore TM Inference Processing Unit (IPU) from Intel Corp (e.g., "Lake Crest") processor. In at least one embodiment, Fig. 7A The illustrated inference and / or training logic 715 may be used in conjunction with central processing unit (“CPU”) hardware, graphics processing unit (“GPU”) hardware, or other hardware such as a field programmable gate array (“FPGA”).

[0087] Figure 7B Inference and / or training logic 715 is shown in accordance with at least one various embodiment. In at least one embodiment, the reasoning and / or training logic 715 may include, but is not limited to, hardware logic in which computing resources are dedicated or otherwise uniquely used in conjunction with weight values ​​or other information corresponding to one or more layers of neurons within a neural network. In at least one embodiment, Figure 7B The inference and / or training logic 715 shown in FIG. 7 may be used in conjunction with an application specific integrated circuit (ASIC), such as the ASIC from Google. Processing unit from Graphcore TM Inference Processing Unit (IPU) from Intel Corp (e.g., "Lake Crest") processor. In at least one embodiment, Figure 7BThe inference and / or training logic 715 shown in can be used in conjunction with central processing unit (CPU) hardware, graphics processing unit (GPU) hardware, or other hardware (e.g., field programmable gate array (FPGA)). In at least one embodiment, the inference and / or training logic 715 includes, but is not limited to, code and / or data storage 701 and code and / or data storage 705, which can be used to store code (e.g., graphics code), weight values ​​and / or other information, including bias values, gradient information, momentum values ​​and / or other parameter or hyperparameter information. Figure 7B In at least one embodiment shown in , each of code and / or data storage 701 and code and / or data storage 705 is associated with a dedicated computing resource (e.g., computing hardware 702 and computing hardware 706), respectively. In at least one embodiment, each of computing hardware 702 and computing hardware 706 includes one or more ALUs that perform mathematical functions (e.g., linear algebraic functions) solely on the information stored in code and / or data storage 701 and code and / or data storage 705, respectively, and the results of performing the functions are stored in activation storage 720.

[0088] In at least one embodiment, each of the code and / or data stores 701 and 705 and the corresponding computing hardware 702 and 706 corresponds to a different layer of the neural network, such that activations from one "storage / compute pair 701 / 702" of the code and / or data store 701 and computing hardware 702 are provided as inputs to the next "storage / compute pair 705 / 706" of the code and / or data store 705 and computing hardware 706, so as to reflect the conceptual organization of the neural network. In at least one embodiment, each storage / compute pair 701 / 702 and 705 / 706 can correspond to more than one neural network layer. In at least one embodiment, additional storage / compute pairs (not shown) can be included in the inference and / or training logic 715 after or in parallel with the storage / compute pairs 701 / 702 and 705 / 706.

[0089] Data Center

[0090] Figure 8 An example data center 800 is shown in which at least one embodiment may be used. In at least one embodiment, the data center 800 includes a data center infrastructure layer 810, a framework layer 820, a software layer 830, and an application layer 840.

[0091] In at least one embodiment, Figure 8As shown, the data center infrastructure layer 810 may include a resource coordinator 812, grouped computing resources 814, and node computing resources ("node CRs") 816(1)-816(N), where "N" represents any complete positive integer. In at least one embodiment, the node CRs 816(1)-816(N) may include, but are not limited to, any number of central processing units ("CPUs") or other processors (including accelerators, field programmable gate arrays (FPGAs), graphics processors, etc.), memory devices (e.g., dynamic read-only memories), storage devices (e.g., solid-state drives or disk drives), network input / output ("NW I / O") devices, network switches, virtual machines ("VMs"), power modules and cooling modules, etc. In at least one embodiment, one or more of the node CRs 816(1)-816(N) may be a server having one or more of the above computing resources.

[0092] In at least one embodiment, the grouped computing resources 814 may include a separate grouping of node CRs housed in one or more racks (not shown), or many racks (also not shown) housed in data centers at various geographic locations. The separate grouping of node CRs within the grouped computing resources 814 may include computing, network, memory or storage resources that can be configured or allocated to support groupings of one or more workloads. In at least one embodiment, several node CRs including a CPU or processor may be grouped in one or more racks to provide computing resources to support one or more workloads. In at least one embodiment, one or more racks may also include any number of power modules, cooling modules and network switches in any combination.

[0093] In at least one embodiment, resource coordinator 812 may configure or otherwise control one or more node CRs 816(1)-816(N) and / or grouped computing resources 814. In at least one embodiment, resource coordinator 812 may include a software design infrastructure ("SDI") management entity for data center 800. In at least one embodiment, resource coordinator 812 may include hardware, software, or some combination thereof.

[0094] In at least one embodiment, Figure 8As shown, the framework layer 820 includes a job scheduler 822, a configuration manager 824, a resource manager 826, and a distributed file system 828. In at least one embodiment, the framework layer 820 may include a framework that supports software 832 of the software layer 830 and / or one or more applications 842 of the application layer 840. In at least one embodiment, the software 832 or the application 842 may include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. In at least one embodiment, the framework layer 820 may be, but is not limited to, a free and open source software web application framework, such as Apache Spark, which may utilize the distributed file system 828 for large-scale data processing (e.g., "big data"). TM (hereinafter referred to as "Spark"). In at least one embodiment, the job scheduler 822 may include a Spark driver to facilitate scheduling of workloads supported by various layers of the data center 800. In at least one embodiment, the configuration manager 824 may be able to configure different layers, such as the software layer 830 and the framework layer 820 including Spark and a distributed file system 828 for supporting large-scale data processing. In at least one embodiment, the resource manager 826 can manage cluster or group computing resources mapped to or allocated to support the distributed file system 828 and the job scheduler 822. In at least one embodiment, the cluster or group computing resources may include grouped computing resources 814 on the data center infrastructure layer 810. In at least one embodiment, the resource manager 826 may coordinate with the resource coordinator 812 to manage these mapped or allocated computing resources.

[0095] In at least one embodiment, software 832 included in software layer 830 may include software used by at least a portion of node CRs 816(1)-816(N), grouped computing resources 814, and / or distributed file system 828 of framework layer 820. One or more types of software may include, but are not limited to, Internet web page search software, email virus scanning software, database software, and streaming video content software.

[0096] In at least one embodiment, one or more applications 842 included in the application layer 840 may include one or more types of applications used by at least a portion of the node CRs 816(1)-816(N), the grouped computing resources 814, and / or the distributed file system 828 of the framework layer 820. The one or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), or other machine learning applications used in conjunction with one or more embodiments.

[0097] In at least one embodiment, any of the configuration manager 824, resource manager 826, and resource coordinator 812 can implement any number and type of self-modification actions based on any number and type of data acquired in any technically feasible manner. In at least one embodiment, the self-modification actions can relieve a data center operator of the data center 800 from making potentially bad configuration decisions and can avoid underutilized and / or poorly performing portions of the data center.

[0098] In at least one embodiment, the data center 800 may include tools, services, software, or other resources to train one or more machine learning models or use one or more machine learning models to predict or infer information according to one or more embodiments described herein. For example, in at least one embodiment, a machine learning model may be trained by calculating weight parameters according to a neural network architecture using the software and computing resources described above with respect to the data center 800. In at least one embodiment, by using weight parameters calculated by one or more training techniques described herein, information may be inferred or predicted using trained machine learning models corresponding to one or more neural networks using the resources described above with respect to the data center 800.

[0099] In at least one embodiment, the data center can use a CPU, an application-specific integrated circuit (ASIC), a GPU, an FPGA, or other hardware to use the above resources to perform training and / or reasoning. In addition, one or more of the above software and / or hardware resources can be configured as a service to allow users to train or perform information reasoning, such as image recognition, speech recognition, or other artificial intelligence services.

[0100] The reasoning and / or training logic 715 is used to perform reasoning and / or training operations associated with one or more embodiments. Fig. 7A and / or Figure 7BProvides details about the reasoning and / or training logic 715. In at least one embodiment, the reasoning and / or training logic 715 may be implemented in the system Figure 8 In some embodiments, the present invention provides a method for performing inference or prediction operations based at least in part on weight parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0101] Such components may be used to generate various scene graphs from one or more rule sets, which may be used to generate training data or image content representing one or more scenes of a virtual environment.

[0102] Computer Systems

[0103] Fig. 9 900, which may include execution units to execute instructions. In at least one embodiment, in accordance with the present disclosure, such as the embodiments described herein, the computer system 900 may include, but is not limited to, components, such as a processor 902, whose execution units include logic to execute algorithms for process data. In at least one embodiment, the computer system 900 may include a processor, such as a processor 902 available from Intel Corporation of Santa Clara, California. Processor family, XeonTM, XScaleTM and / or StrongARMTM, Core TM or Nervana TM microprocessor, although other systems (including PCs with other microprocessors, engineering workstations, set-top boxes, etc.) may also be used. In at least one embodiment, computer system 900 may execute a version of the WINDOWS operating system available from Microsoft Corporation of Redmond, Wash., although other operating systems (e.g., UNIX and Linux), embedded software, and / or graphical user interfaces may also be used.

[0104] Embodiments may be used in other devices, such as handheld devices and embedded applications. Some examples of handheld devices include cellular phones, Internet Protocol (Internet Protocol) devices, digital cameras, personal digital assistants ("PDAs"), and handheld PCs. In at least one embodiment, the embedded application may include a microcontroller, a digital signal processor ("DSP"), a system on a chip, a network computer ("NetPC"), a set-top box, a network hub, a wide area network ("WAN") switch, or any other system that can execute one or more instructions according to at least one embodiment.

[0105] In at least one embodiment, the computer system 900 may include, but is not limited to, a processor 902, which may include, but is not limited to, one or more execution units 908 to perform machine learning model training and / or reasoning according to the techniques described herein. In at least one embodiment, the computer system 900 is a single-processor desktop or server system, but in another embodiment, the computer system 900 may be a multi-processor system. In at least one embodiment, the processor 902 may include, but is not limited to, a complex instruction set computer ("CISC") microprocessor, a reduced instruction set computing ("RISC") microprocessor, a very long instruction word ("VLIW") microprocessor, a processor that implements an instruction set combination, or any other processor device, such as a digital signal processor. In at least one embodiment, the processor 902 may be coupled to a processor bus 910, which may transmit data signals between the processor 902 and other components in the computer system 900.

[0106] In at least one embodiment, processor 902 may include, but is not limited to, a level 1 ("L1") internal cache memory ("cache") 904. In at least one embodiment, processor 902 may have a single internal cache or multiple levels of internal cache. In at least one embodiment, the cache memory may reside external to processor 902. Other embodiments may also include a combination of internal and external caches, depending on the particular implementation and needs. In at least one embodiment, register file 906 may store different types of data in various registers, including, but not limited to, integer registers, floating point registers, status registers, and instruction pointer registers.

[0107] In at least one embodiment, an execution unit 908, including but not limited to logic to perform integer and floating point operations, is also located in the processor 902. In at least one embodiment, the processor 902 may also include a microcode ("ucode") read-only memory ("ROM") for storing microcode for certain macroinstructions. In at least one embodiment, the execution unit 908 may include logic for processing a packed instruction set 909. In at least one embodiment, by including the packed instruction set 909 in the instruction set of the general purpose processor 902, and the associated circuitry to execute the instructions, operations used by many multimedia applications may be performed using packed data in the general purpose processor 902. In one or more embodiments, many multimedia applications may be executed faster and more efficiently by using the full width of the processor's data bus to perform operations on the packed data, which may not require the transfer of smaller units of data on the processor's data bus to perform one or more operations one data element at a time.

[0108] In at least one embodiment, execution unit 908 may also be used in microcontrollers, embedded processors, graphics devices, DSPs, and other types of logic circuits. In at least one embodiment, computer system 900 may include, but is not limited to, memory 920. In at least one embodiment, memory 920 may be implemented as a dynamic random access memory ("DRAM") device, a static random access memory ("SRAM") device, a flash memory device, or other storage device. In at least one embodiment, memory 920 may store instructions 919 and / or data 921 represented by data signals that may be executed by processor 902.

[0109] In at least one embodiment, the system logic chip can be coupled to the processor bus 910 and the memory 920. In at least one embodiment, the system logic chip can include, but is not limited to, a memory controller hub ("MCH") 916, and the processor 902 can communicate with the MCH 916 via the processor bus 910. In at least one embodiment, the MCH 916 can provide a high-bandwidth memory path 918 to the memory 920 for instruction and data storage and for storage of graphics commands, data, and textures. In at least one embodiment, the MCH 916 can initiate data signals between the processor 902, the memory 920, and other components in the computer system 900, and bridge data signals between the processor bus 910, the memory 920, and the system I / O 922. In at least one embodiment, the system logic chip can provide a graphics port for coupling to a graphics controller. In at least one embodiment, the MCH 916 can be coupled to the memory 920 via a high-bandwidth memory path 918, and the graphics / video card 912 can be coupled to the MCH 916 via an Accelerated Graphics Port ("AGP") interconnect 914.

[0110] In at least one embodiment, the computer system 900 can use the system I / O 922 as a proprietary hub interface bus to couple the MCH 916 to an I / O controller hub ("ICH") 930. In at least one embodiment, the ICH 930 can provide direct connection to certain I / O devices through a local I / O bus. In at least one embodiment, the local I / O bus can include, but is not limited to, a high-speed I / O bus used to connect peripheral devices to the memory 920, the chipset, and the processor 902. Examples can include, but are not limited to, an audio controller 929, a firmware hub ("Flash BIOS") 928, a wireless transceiver 926, a data store 924, a traditional I / O controller 923 including user input and a keyboard interface 925, a serial expansion port 927 (e.g., a universal serial bus (USB)), and a network controller 934. The data store 924 can include a hard drive, a floppy drive, a CD-ROM device, a flash memory device, or other mass storage device.

[0111] In at least one embodiment, Fig. 9 A system comprising interconnected hardware devices or "chips" is shown, while in other embodiments, Fig. 9 An exemplary system on a chip ("SoC") may be shown. In at least one embodiment, the devices may be interconnected with a proprietary interconnect, a standardized interconnect (e.g., PCIe), or some combination thereof. In at least one embodiment, one or more components of computer system 900 are interconnected using a Compute Express Link (CXL) interconnect.

[0112] The reasoning and / or training logic 715 is used to perform reasoning and / or training operations related to one or more embodiments. Fig. 7A and / or Figure 7B Provide details about the reasoning and / or training logic 715. In at least one embodiment, the reasoning and / or training logic 715 may be Fig. 9 for use in a system for performing inference or prediction operations based at least in part on weight parameters computed using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0113] Such components may be used to generate various scene graphs from one or more rule sets, which may be used to generate training data or image content representing one or more scenes of a virtual environment.

[0114] Fig.10 1 is a block diagram illustrating an electronic device 1000 for utilizing a processor 1010 according to at least one embodiment. In at least one embodiment, the electronic device 1000 may be, for example but not limited to, a notebook computer, a tower server, a rack server, a blade server, a laptop computer, a desktop computer, a tablet computer, a mobile device, a phone, an embedded computer, or any other suitable electronic device.

[0115] In at least one embodiment, system 1000 may include, but is not limited to, a processor 1010 communicatively coupled to any suitable number or kind of components, peripherals, modules, or devices. In at least one embodiment, processor 1010 is coupled using a bus or interface, such as an I2C bus, a system management bus ("SMBus"), a low pin count (LPC) bus, a serial peripheral interface ("SPI"), a high-definition audio ("HDA") bus, a serial advanced technology attachment ("SATA") bus, a universal serial bus ("USB") (versions 1, 2, 3), or a universal asynchronous receiver / transmitter ("UART") bus. In at least one embodiment, Fig.10 15 illustrates a system including interconnected hardware devices or "chips," while in other embodiments, FIG. 15 may illustrate an exemplary system on a chip ("SoC"). In at least one embodiment, Fig.10 The devices shown in can be interconnected with a proprietary interconnect, a standardized interconnect (e.g., PCIe), or some combination thereof. In at least one embodiment, Fig.10 One or more components of the system are interconnected using Compute Express Link (CXL) interconnect lines.

[0116] In at least one embodiment, Fig.10The system may include a display 1024, a touch screen 1025, a touch pad 1030, a near field communication unit (“NFC”) 1045, a sensor hub 1040, a thermal sensor 1046, a fast chipset (“EC”) 1035, a trusted platform module (“TPM”) 1038, a BIOS / firmware / flash memory (“BIOS, FW Flash”) 1022, a DSP 1060, a drive (“SSD” or “HDD”) 1020 (e.g., a solid state disk (“SSD”) or a hard disk drive (“HDD”)), a wireless local area network unit (“WLAN”) 1050, a Bluetooth unit 1052, a wireless wide area network unit (“WWAN”) 1056, a global positioning system (GPS) 1055, and a camera (“USB 3.0 camera”) 1054 (e.g., a USB 3.0 camera). 3.0 camera) and / or a low power double data rate ("LPDDR") memory unit ("LPDDR3") 1015 implemented in, for example, the LPDDR3 standard. Each of these components may be implemented in any suitable manner.

[0117] In at least one embodiment, other components may be communicatively coupled to the processor 1010 through the components discussed above. In at least one embodiment, the accelerometer 1041, ambient light sensor (“ALS”) 1042, compass 1043, and gyroscope 1044 may be communicatively coupled to the sensor hub 1040. In at least one embodiment, the thermal sensor 1039, fan 1037, keyboard 1046, and touchpad 1030 may be communicatively coupled to the EC 1035. In at least one embodiment, the speaker 1063, earphone 1064, and microphone (“mic”) 1065 may be communicatively coupled to the audio unit (“audio codec and class D amplifier”) 1064, which in turn may be communicatively coupled to the DSP 1060. In at least one embodiment, the audio unit 1064 may include, for example, but not limited to, an audio encoder / decoder (“codec”) and a class D amplifier. In at least one embodiment, the SIM card (“SIM”) 1057 may be communicatively coupled to the WWAN unit 1056. In at least one embodiment, components such as the WLAN unit 1050 and the Bluetooth unit 1052 and the WWAN unit 1056 may be implemented as a next generation form factor ("NGFF").

[0118] The reasoning and / or training logic 715 is used to perform reasoning and / or training operations associated with one or more embodiments. Fig. 7A and / or Figure 7B Provides details about the reasoning and / or training logic 715. In at least one embodiment, the reasoning and / or training logic 1015 may be in the system Fig.10for use in reasoning or predicting operations based at least in part on weight parameters computed using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0119] Such components may be used to generate various scene graphs from one or more rule sets, which may be used to generate training data or image content representing one or more scenes of a virtual environment.

[0120] Fig.11 1100 is a block diagram of a processing system according to at least one embodiment. In at least one embodiment, system 1100 includes one or more processors 1102 and one or more graphics processors 1108, and can be a single processor desktop system, a multi-processor workstation system, or a server system with a large number of processors 1102 or processor cores 1107. In at least one embodiment, system 1100 is a processing platform incorporated in a system-on-chip (SoC) integrated circuit for use in a mobile, handheld, or embedded device.

[0121] In at least one embodiment, the system 1100 may include or be incorporated into a server-based gaming platform, a gaming console including a gaming and media console, a mobile gaming console, a handheld gaming console, or an online gaming console. In at least one embodiment, the system 1100 is a mobile phone, a smart phone, a tablet computing device, or a mobile Internet device. In at least one embodiment, the processing system 1100 may also include a wearable device, coupled to the wearable device or integrated into the wearable device, such as a smart watch wearable device, a smart glasses device, an augmented reality device, or a virtual reality device. In at least one embodiment, the processing system 1100 is a television or set-top box device having one or more processors 1102 and a graphical interface generated by one or more graphics processors 1108.

[0122] In at least one embodiment, one or more processors 1102 each include one or more processor cores 1107 to process instructions that, when executed, perform operations for system and user software. In at least one embodiment, each of the one or more processor cores 1107 is configured to process a specific instruction set 1109. In at least one embodiment, the instruction set 1109 can facilitate complex instruction set computing (CISC), reduced instruction set computing (RISC), or computing via very long instruction words (VLIW). In at least one embodiment, the processor cores 1107 can each process different instruction sets 1109, which may include instructions that help emulate other instruction sets. In at least one embodiment, the processor cores 1107 may also include other processing devices, such as digital signal processors (DSPs).

[0123] In at least one embodiment, the processor 1102 includes a cache memory 1104. In at least one embodiment, the processor 1102 may have a single internal cache or multiple levels of internal cache. In at least one embodiment, the cache memory is shared between various components of the processor 1102. In at least one embodiment, the processor 1102 also uses an external cache (e.g., a level 3 (L3) cache or a last level cache (LLC)) (not shown), which can be shared between the processor cores 1107 using known cache coherence techniques. In at least one embodiment, the processor 1102 additionally includes a register file 1106, which may include different types of registers (e.g., integer registers, floating point registers, status registers, and instruction pointer registers) for storing different types of data. In at least one embodiment, the register file 1106 may include general registers or other registers.

[0124] In at least one embodiment, one or more processors 1102 are coupled to one or more interface buses 1110 to transmit communication signals, such as address, data, or control signals, between the processor 1102 and other components in the system 1100. In at least one embodiment, the interface bus 1110 in one embodiment may be a processor bus, such as a version of a direct media interface (DMI) bus. In at least one embodiment, the interface 1110 is not limited to a DMI bus, and may include one or more peripheral component interconnect buses (e.g., PCI, PCI Express), memory buses, or other types of interface buses. In at least one embodiment, the processor 1102 includes an integrated memory controller 1116 and a platform controller hub 1130. In at least one embodiment, the memory controller 1116 facilitates communication between memory devices and other components of the system 1100, while the platform controller hub (PCH) 1130 provides connections to I / O devices via a local I / O bus.

[0125] In at least one embodiment, the memory device 1120 may be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, a flash memory device, a phase change memory device, or other memory device with appropriate performance for use as process memory. In at least one embodiment, the memory device 1120 may be used as system memory for the system 1100 to store data 1122 and instructions 1121 for use when one or more processors 1102 execute applications or processes. In at least one embodiment, the memory controller 1116 is also coupled to an optional external graphics processor 1112, which may communicate with one or more graphics processors 1108 in the processors 1102 to perform graphics and media operations. In at least one embodiment, a display device 1111 may be connected to one or more processors 1102. In at least one embodiment, the display device 1111 may include one or more of the internal display devices, such as in a mobile electronic device or laptop device or an external display device connected via a display interface (e.g., DisplayPort, etc.). In at least one embodiment, the display device 1111 may include a head mounted display (HMD), such as a stereoscopic display device used in virtual reality (VR) applications or augmented reality (AR) applications.

[0126] In at least one embodiment, the platform controller hub 1130 enables peripheral devices to be connected to the memory device 1120 and the processor 1102 via a high-speed I / O bus. In at least one embodiment, the I / O peripherals include, but are not limited to, an audio controller 1146, a network controller 1134, a firmware interface 1128, a wireless transceiver 1126, a touch sensor 1125, a data storage device 1124 (e.g., a hard drive, flash memory, etc.). In at least one embodiment, the data storage device 1124 can be connected via a storage interface (e.g., SATA) or via a peripheral bus such as a peripheral component interconnect bus (e.g., PCI, PCI Express). In at least one embodiment, the touch sensor 1125 can include a touch screen sensor, a pressure sensor, or a fingerprint sensor. In at least one embodiment, the wireless transceiver 1126 can be a Wi-Fi transceiver, a Bluetooth transceiver, or a mobile network transceiver such as a 3G, 4G, or long-term evolution (LTE) transceiver. In at least one embodiment, the firmware interface 1128 enables communication with the system firmware and can be, for example, a unified extensible firmware interface (UEFI). In at least one embodiment, the network controller 1134 can enable a network connection to a wired network. In at least one embodiment, a high-performance network controller (not shown) is coupled to the interface bus 1110. In at least one embodiment, the audio controller 1146 is a multi-channel high-definition audio controller. In at least one embodiment, the system 1100 includes an optional legacy I / O controller 1140 for coupling legacy (e.g., Personal System 2 (PS / 2)) devices to the system. In at least one embodiment, the platform controller hub 1130 can also be connected to one or more universal serial bus (USB) controllers 1142, which connect input devices such as a keyboard and mouse 1143 combination, a camera 114, or other USB input devices.

[0127] In at least one embodiment, instances of memory controller 1116 and platform controller hub 1130 may be integrated into a discrete external graphics processor, such as external graphics processor 1112. In at least one embodiment, platform controller hub 1130 and / or memory controller 1116 may be external to one or more processors 1102. For example, in at least one embodiment, system 1100 may include external memory controller 1116 and platform controller hub 1130, which may be configured as a memory controller hub and a peripheral controller hub in a system chipset that communicates with one or more processors 1102.

[0128] The reasoning and / or training logic 715 is used to perform reasoning and / or training operations associated with one or more embodiments. Fig. 7A and / or Figure 7BDetail is provided regarding the inference and / or training logic 715. In at least one embodiment, some or all of the inference and / or training logic 715 may be incorporated into the graphics processor 1500. For example, in at least one embodiment, the training and / or inference techniques described herein may use one or more ALUs embodied in the graphics processor. In addition, in at least one embodiment, the inference and / or training operations described herein may use the inference and / or training logic 715. Fig. 7A or Figure 7B In at least one embodiment, the weight parameters may be stored in on-chip or off-chip memory and / or registers (shown or not shown) that configure the ALU of the graphics processor to execute one or more machine learning algorithms, neural network architectures, use cases, or training techniques described herein.

[0129] Such components may be used to generate various scene graphs from one or more rule sets, which may be used to generate training data or image content representing one or more scenes of a virtual environment.

[0130] Fig.12 is a block diagram of a processor 1200 having one or more processor cores 1202A-1202N, an integrated memory controller 1214, and an integrated graphics processor 1208 in accordance with at least one embodiment. In at least one embodiment, the processor 1200 may include additional cores up to and including the additional core 1202N represented by the dashed box. In at least one embodiment, each processor core 1202A-1202N includes one or more internal cache units 1204A-1204N. In at least one embodiment, each processor core may also have access to one or more shared cache units 1206.

[0131] In at least one embodiment, the internal cache units 1204A-1204N and the shared cache unit 1206 represent a cache memory hierarchy within the processor 1200. In at least one embodiment, the cache memory units 1204A-1204N may include at least one level of instruction and data cache in one or more levels of each processor core and a shared mid-level cache, such as level 2 (L2), level 3 (L3), level 4 (L4), or other levels of cache, where the highest level of cache before external memory is classified as LLC. In at least one embodiment, cache coherency logic maintains coherency between the various cache units 1206 and 1204A-1204N.

[0132] In at least one embodiment, the processor 1200 may also include a set of one or more bus controller units 1216 and a system agent core 1210. In at least one embodiment, the one or more bus controller units 1216 manage a set of peripheral buses, such as one or more PCI or PCI Express buses. In at least one embodiment, the system agent core 1210 provides management functions for various processor components. In at least one embodiment, the system agent core 1210 includes one or more integrated memory controllers 1214 to manage access to various external memory devices (not shown).

[0133] In at least one embodiment, one or more processor cores 1202A-1202N include support for simultaneous multithreading. In at least one embodiment, system agent core 1210 includes components for coordinating and operating cores 1202A-1202N during multithreaded processing. In at least one embodiment, system agent core 1210 may additionally include a power control unit (PCU) that includes logic and components to regulate one or more power states of processor cores 1202A-1202N and graphics processor 1208.

[0134] In at least one embodiment, the processor 1200 additionally includes a graphics processor 1208 to perform graphics processing operations. In at least one embodiment, the graphics processor 1208 is coupled to a shared cache unit 1206 and a system agent core 1210 including one or more integrated memory controllers 1214. In at least one embodiment, the system agent core 1210 also includes a graphics processor for a display controller 1211 to drive output to one or more coupled displays. In at least one embodiment, the display controller 1211 may also be a separate module coupled to the graphics processor 1208 via at least one interconnect, or may be integrated within the graphics processor 1208.

[0135] In at least one embodiment, a ring-based interconnect unit 1212 is used to couple the internal components of the processor 1200. In at least one embodiment, alternative interconnect units may be used, such as point-to-point interconnects, switched interconnects, or other technologies. In at least one embodiment, the graphics processor 1208 is coupled to the ring interconnect 1212 via an I / O link 1213.

[0136] In at least one embodiment, I / O link 1213 represents at least one of a variety of I / O interconnects, including an on-package I / O interconnect, such as an eDRAM module, that facilitates communication between various processor components and high-performance embedded memory module 1218. In at least one embodiment, each of processor cores 1202A-1202N and graphics processor 1208 uses embedded memory module 1218 as a shared last level cache.

[0137] In at least one embodiment, the processor cores 1202A-1202N are homogeneous cores that execute a common instruction set architecture. In at least one embodiment, the processor cores 1202A-1202N are heterogeneous in terms of instruction set architecture (ISA), wherein one or more of the processor cores 1202A-1202N execute a common instruction set, while one or more other cores of the processor cores 1202A-1202N execute a subset of a common instruction set or a different instruction set. In at least one embodiment, the processor cores 1202A-1202N are heterogeneous in terms of microarchitecture, wherein one or more cores with relatively high power consumption are coupled with one or more power cores with lower power consumption. In at least one embodiment, the processor 1200 may be implemented on one or more chips or as a SoC integrated circuit.

[0138] The reasoning and / or training logic 715 is used to perform reasoning and / or training operations associated with one or more embodiments. Details regarding the reasoning and / or training logic 715 are provided below in conjunction with FIG. 7a and / or FIG. 7b. In at least one embodiment, part or all of the reasoning and / or training logic 715 may be incorporated into the processor 1200. For example, in at least one embodiment, the training and / or reasoning techniques described herein may be used in the graphics processor 1512, one or more graphics cores 1202A-1202N, or Fig.12 In addition, in at least one embodiment, the inference and / or training operations described herein may use the addition Fig. 7A or Figure 7B In at least one embodiment, the weight parameters may be stored in on-chip or off-chip memory and / or registers (shown or not shown) that configure the ALU of the graphics processor 1200 to execute one or more machine learning algorithms, neural network architectures, use cases, or training techniques described herein.

[0139] Such components may be used to generate various scene graphs from one or more rule sets, which may be used to generate training data or image content representing one or more scenes of a virtual environment.

[0140] Virtualized computing platform

[0141] Fig.13 is an example data flow diagram of a process 1300 for generating and deploying an image processing and inference pipeline according to at least one embodiment. In at least one embodiment, the process 1300 can be deployed for use with imaging devices, processing devices, and / or other device types at one or more facilities 1302. The process 1300 can be executed within a training system 1304 and / or a deployment system 1306. In at least one embodiment, the training system 1304 can be used to perform training, deployment, and implementation of machine learning models (e.g., neural networks, object detection algorithms, computer vision algorithms, etc.) for use with the deployment system 1306. In at least one embodiment, the deployment system 1306 can be configured to offload processing and computing resources in a distributed computing environment to reduce infrastructure requirements for the facility 1302. In at least one embodiment, one or more applications in the pipeline can use or call services (e.g., reasoning, visualization, computation, AI, etc.) of the deployment system 1306 during execution of the application.

[0142] In at least one embodiment, some applications used in the high-level processing and reasoning pipeline may use machine learning models or other AI to perform one or more processing steps. In at least one embodiment, the machine learning model may be trained at the facility 1302 using data 1308 (e.g., imaging data) generated at the facility 1302 (and the data 1308 is stored on one or more picture archiving and communication system (PACS) servers at the facility 1302), trained using imaging or sequencing data 1308 from another facility, or a combination thereof. In at least one embodiment, the training system 1304 may be used to provide applications, services, and / or other resources to generate deployable machine learning models for use in the work of the deployment system 1306.

[0143] In at least one embodiment, model registry 1324 can be backed by an object store that can support version control and object metadata. In at least one embodiment, the model registry 1324 can be backed by an object store that can support version control and object metadata. Fig.14 The object store is accessed through an application programming interface (API) compatible with the cloud 1426 of the system. In at least one embodiment, the machine learning models within the model registry 1324 can be uploaded, listed, modified, or deleted by personnel or partners of the system interacting with the API. In at least one embodiment, the API can provide access to a method that allows a user with appropriate credentials to associate a model with an application so that the model can be executed as part of the execution of a containerized instance of the application.

[0144] In at least one embodiment, training pipeline 1404 ( Fig.14 ) may include scenarios where the facility 1302 is training their own machine learning models, or has an existing machine learning model that needs to be optimized or updated. In at least one embodiment, imaging data 1308 generated by one or more imaging devices, sequencing devices, and / or other device types may be received. In at least one embodiment, once the imaging data 1308 is received, the AI-assisted annotation 1310 may be used to help generate annotations corresponding to the imaging data 1308 to be used as ground truth data for the machine learning model. In at least one embodiment, the AI-assisted annotation 1310 may include one or more machine learning models (e.g., convolutional neural networks (CNNs)) that may be trained to generate annotations corresponding to certain types of imaging data 1038 (e.g., from certain devices). In at least one embodiment, the AI-assisted annotation 1310 may then be used directly, or may be adjusted or fine-tuned using an annotation tool to generate ground truth data. In at least one embodiment, the AI-assisted annotation 1310, the labeled clinical data 1312, or a combination thereof may be used as ground truth data for training a machine learning model. In at least one embodiment, the trained machine learning model can be referred to as an output model 1316 and can be used by the deployment system 1306 as described herein.

[0145] In at least one embodiment, training pipeline 1404 ( Fig.14) may include scenarios where facility 1302 requires a machine learning model for performing one or more processing tasks of one or more applications in deployment system 1306, but facility 1302 may not currently have such a machine learning model (or may not have an efficient or effective model optimized for this purpose). In at least one embodiment, an existing machine learning model may be selected from model registry 1324. In at least one embodiment, model registry 1324 may include machine learning models that have been trained to perform a variety of different reasoning tasks on imaging data. In at least one embodiment, the machine learning models in model registry 1324 may be trained on imaging data from a facility different from facility 1302 (e.g., a remotely located facility). In at least one embodiment, the machine learning model may have been trained on imaging data from one location, two locations, or any number of locations. In at least one embodiment, when training on imaging data from a particular location, the training may be performed at that location, or at least in a manner that protects the confidentiality of the imaging data or restricts the transfer of the imaging data from the local area. In at least one embodiment, once a model is trained or partially trained at one location, the machine learning model may be added to model registry 1324. In at least one embodiment, the machine learning model can then be retrained or updated at any number of other facilities, and the retrained or updated model can be made available in the model registry 1324. In at least one embodiment, the machine learning model can then be selected from the model registry 1324, and referred to as the output model 1316, and can be used in the deployment system 1306 to perform one or more processing tasks for one or more applications of the deployment system.

[0146] In at least one embodiment, training pipeline 1404 ( Fig.14), the scenario may include a facility 1302 that requires a machine learning model for performing one or more processing tasks for one or more applications in the deployment system 1306, but the facility 1302 may not currently have such a machine learning model (or may not have an efficient or effective model optimized for this purpose). In at least one embodiment, due to population differences, robustness of the training data used to train the machine learning model, diversity of training data anomalies, and / or other issues with the training data, the machine learning model selected from the model registry 1324 may not be fine-tuned or optimized for the imaging data 1308 generated at the facility 1302. In at least one embodiment, AI-assisted annotations 1310 can be used to assist in generating annotations corresponding to the imaging data 1308 for use as ground truth data for retraining or updating the machine learning model. In at least one embodiment, labeled data 1312 can be used as ground truth data for training the machine learning model. In at least one embodiment, retraining or updating the machine learning model can be referred to as model training 1314. In at least one embodiment, model training 1314 (e.g., AI-assisted annotations 1310, labeled clinical data 1312, or a combination thereof) can be used as ground truth data to retrain or update the machine learning model. In at least one embodiment, the trained machine learning model can be referred to as an output model 1316 and can be used by the deployment system 1306, as described herein.

[0147] In at least one embodiment, the deployment system 1306 may include software 1318, services 1320, hardware 1322, and / or other components, features, and functions. In at least one embodiment, the deployment system 1306 may include a software "stack" such that the software 1318 may be built on top of the services 1320 and may use the services 1320 to perform some or all processing tasks, and the services 1320 and software 1318 may be built on top of the hardware 1322 and use the hardware 1322 to perform processing, storage, and / or other computing tasks of the deployment system 1306. In at least one embodiment, the software 1318 may include any number of different containers, each of which may perform an instantiation of an application. In at least one embodiment, each application may perform one or more processing tasks (e.g., reasoning, object detection, feature detection, segmentation, image enhancement, calibration, etc.) in a high-level processing and reasoning pipeline. In at least one embodiment, in addition to receiving and configuring containers for imaging data for use by each container and / or device 1302 after processing through the pipeline (e.g., to convert the output back into a usable data type), a high-level processing and reasoning pipeline can be defined based on the selection of different containers desired or required to process imaging data 1308. In at least one embodiment, a combination of containers within software 1318 (e.g., containers that make up a pipeline) can be referred to as a virtual instrument (as described in more detail herein), and the virtual instrument can utilize services 1320 and hardware 1322 to perform some or all of the processing tasks of an application instantiated in a container.

[0148] In at least one embodiment, the data processing pipeline may receive input data (e.g., imaging data 1308) in a particular format in response to an inference request (e.g., a request from a user of the deployment system 1306). In at least one embodiment, the input data may represent one or more images, videos, and / or other data representations generated by one or more imaging devices. In at least one embodiment, the data may be pre-processed as part of the data processing pipeline to prepare the data for processing by one or more applications. In at least one embodiment, post-processing may be performed on the output of one or more inference tasks or other processing tasks of the pipeline to prepare the output data for the next application and / or to prepare the output data for transmission and / or user use (e.g., as a response to the inference request). In at least one embodiment, the inference task may be performed by one or more machine learning models, such as a trained or deployed neural network, which may include the output model 1316 of the training system 1304.

[0149] In at least one embodiment, the tasks of a data processing pipeline may be encapsulated in one or more containers, each container representing a discrete, fully functional instantiation of an application and a virtualized computing environment capable of referencing a machine learning model. In at least one embodiment, a container or application may be published to a private (e.g., limited access) area of ​​a container registry (described in more detail herein), and a trained or deployed model may be stored in the model registry 1324 and associated with one or more applications. In at least one embodiment, an image of an application (e.g., a container image) may be available in a container registry, and once selected by a user from the container registry for deployment in a pipeline, the image may be used to generate an instantiated container for the application for use by the user's system.

[0150] In at least one embodiment, developers (e.g., software developers, clinicians, doctors, etc.) can develop, publish, and store applications (e.g., as containers) for performing image processing and / or inference on provided data. In at least one embodiment, a software development kit (SDK) associated with the system can be used to perform development, publishing, and / or storage (e.g., to ensure that the developed applications and / or containers conform to or are compatible with the system). In at least one embodiment, a system (e.g., Fig.14 The developed application may be tested locally (e.g., at the first facility, on data from the first facility) using an SDK of the system 1400 of the present invention. In at least one embodiment, because a DICOM object may contain anywhere from one to hundreds of images or other data types, and because the data changes, the developer may be responsible for managing (e.g., setting up constructs to build pre-processing into the application, etc.) the extraction and preparation of the incoming data. In at least one embodiment, once verified by the system 1400 (e.g., for accuracy), the application may be available in a container registry for the user to select and / or implement to perform one or more processing tasks on the data at the user's facility (e.g., the second facility).

[0151] In at least one embodiment, the person can then share the application or container over the network for use by a system (e.g., Fig.141400). In at least one embodiment, the completed and validated application or container can be stored in the container registry, and the associated machine learning model can be stored in the model registry 1324. In at least one embodiment, the requesting entity (which provides an inference or image processing request) can browse the container registry and / or the model registry 1324 to find applications, containers, data sets, machine learning models, etc., select the desired combination of elements to be included in the data processing pipeline, and submit the imaging processing request. In at least one embodiment, the request may include the input data necessary to execute the request (and in some examples, data associated with the patient), and / or may include the selection of one or more applications and / or machine learning models to be executed in the processing request. In at least one embodiment, the request can then be passed to one or more components (e.g., the cloud) of the deployment system 1306 to perform processing of the data processing pipeline. In at least one embodiment, the processing performed by the deployment system 1306 may include referencing the selected elements (e.g., applications, containers, models, etc.) from the container registry and / or the model registry 1324. In at least one embodiment, once the results are generated by the pipeline, they can be returned to the user for reference (e.g., for viewing in a viewing application suite executing locally, on a local workstation or terminal).

[0152] In at least one embodiment, to assist in processing or executing applications or containers in the pipeline, services 1320 may be utilized. In at least one embodiment, services 1320 may include computing services, artificial intelligence (AI) services, visualization services, and / or other service types. In at least one embodiment, services 1320 may provide functionality common to one or more applications in software 1318, and thus may abstract functionality into services that may be called or utilized by applications. In at least one embodiment, the functionality provided by services 1320 may run dynamically and more efficiently, while also enabling applications to process data in parallel (e.g., using parallel computing platform 1430 ( Fig.14)) to scale well. In at least one embodiment, rather than requiring each application that shares the same functionality provided by the service 1320 to have a corresponding instance of the service 1320, the service 1320 can be shared between and among various applications. In at least one embodiment, as a non-limiting example, the service may include an inference server or engine that can be used to perform detection or segmentation tasks. In at least one embodiment, a model training service may be included that can provide machine learning model training and / or retraining capabilities. In at least one embodiment, a data enhancement service may be further included that can provide GPU-accelerated data (e.g., DICOM, RIS, CIS, REST-compatible, RPC, original, etc.) extraction, resizing, scaling, and / or other enhancements. In at least one embodiment, a visualization service may be used that can add image rendering effects (e.g., ray tracing, rasterization, denoising, sharpening, etc.) to add realism to two-dimensional (2D) and / or three-dimensional (3D) model images. In at least one embodiment, a virtual instrument service may be included that provides beamforming, segmentation, reasoning, imaging, and / or support for other applications within the pipeline of the virtual instrument.

[0153] In at least one embodiment, where the service 1320 includes an AI service (e.g., an inference service), one or more machine learning models may be executed by calling (e.g., as an API call) an inference service (e.g., an inference server) to execute one or more machine learning models or process as part of application execution. In at least one embodiment, where another application includes one or more machine learning models for a segmentation task, the application may call the inference service to execute the machine learning model for performing one or more processing operations associated with the segmentation task. In at least one embodiment, software 1318 implementing a high-level processing and inference pipeline including a segmentation application and anomaly detection application may be simplified because each application may call the same inference service to perform one or more inference tasks.

[0154] In at least one embodiment, the hardware 1322 may include a GPU, a CPU, a graphics card, an AI / deep learning system (e.g., an AI supercomputer, such as NVIDIA's DGX), a cloud platform, or a combination thereof. In at least one embodiment, different types of hardware 1322 may be used to provide effective, specific-purpose support for the software 1318 and services 1320 in the deployment system 1306. In at least one embodiment, the use of GPU processing may be implemented for local processing (e.g., at the facility 1302), within the AI / deep learning system, in the cloud system, and / or in other processing components of the deployment system 1306 to improve the efficiency, accuracy, and efficacy of image processing and generation. In at least one embodiment, as a non-limiting example, the software 1318 and / or services 1320 may be optimized for GPU processing for deep learning, machine learning, and / or high-performance computing. In at least one embodiment, at least some of the computing environments of the deployment system 1306 and / or training system 1304 may utilize GPU-optimized software (e.g., a combination of NVIDIA's DGX system hardware and software) to execute in one or more supercomputers or high-performance computing systems in a data center. In at least one embodiment, hardware 1322 may include any number of GPUs that can be called to perform data processing in parallel, as described herein. In at least one embodiment, the cloud platform may also include GPU processing for GPU-optimized execution of deep learning tasks, machine learning tasks, or other computing tasks. In at least one embodiment, a cloud platform (e.g., NVIDIA's NGC) may be executed using one or more AI / deep learning supercomputers and / or GPU-optimized software (e.g., as provided on NVIDIA's DGX system) as a hardware abstraction and scaling platform. In at least one embodiment, the cloud platform can integrate an application container cluster system or orchestration system (e.g., Kubernetes) on multiple GPUs to achieve seamless scaling and load balancing.

[0155] Fig.14 is a system diagram of an example system 1400 for generating and deploying an imaging deployment pipeline according to at least one embodiment. In at least one embodiment, the system 1400 can be used to implement a high-level processing and reasoning pipeline. Fig.13 In at least one embodiment, system 1400 may include training system 1304 and deployment system 1306. In at least one embodiment, training system 1304 and deployment system 1306 may be implemented using software 1318, services 1320, and / or hardware 1322, as described herein.

[0156] In at least one embodiment, system 1400 (e.g., training system 1304 and / or deployment system 1306) can be implemented in a cloud computing environment (e.g., using cloud 1426). In at least one embodiment, system 1400 can be implemented locally with respect to a healthcare service facility, or as a combination of cloud and local computing resources. In at least one embodiment, access to APIs in cloud 1426 can be restricted to authorized users through established security measures or protocols. In at least one embodiment, the security protocol can include a network token that can be signed by an authentication (e.g., AuthN, AuthZ, Gluecon, etc.) service and can carry appropriate authorization. In at least one embodiment, the API of the virtual instrument (described herein) or other instances of system 1400 can be restricted to a set of public IPs that have been audited or authorized for interaction.

[0157] In at least one embodiment, the various components of system 1400 may communicate with each other via wired and / or wireless communication protocols using any of a variety of different network types, including, but not limited to, a local area network (LAN) and / or a wide area network (WAN). In at least one embodiment, communications between facilities and components of system 1400 (e.g., for sending inference requests, for receiving results of inference requests, etc.) may be transmitted over one or more data buses, wireless data protocols (Wi-Fi), wired data protocols (e.g., Ethernet), etc.

[0158] In at least one embodiment, training system 1304 can execute training pipeline 1404, similar to the training pipeline described herein. Fig.13 those described. In at least one embodiment, where one or more machine learning models are to be used in a deployment pipeline 1410 by the deployment system 1306, the training pipeline 1404 may be used to train or retrain one or more (e.g., pre-trained) models, and / or implement one or more pre-trained models 1406 (e.g., without the need for retraining or updating). In at least one embodiment, as a result of the training pipeline 1404, one or more output models 1316 may be generated. In at least one embodiment, the training pipeline 1404 may include any number of processing steps, such as, but not limited to, transformation or adaptation of imaging data (or other input data). In at least one embodiment, different training pipelines 1404 may be used for different machine learning models used by the deployment system 1306. In at least one embodiment, similar to the description regarding Fig.13 The first example training pipeline 1404 described may be used for a first machine learning model, similar to the training pipeline 1404 described with respect to FIG. Fig.13 The second example training pipeline 1404 described can be used for a second machine learning model and is similar to the training pipeline 1404 described with respect to Fig.13The training pipeline 1404 of the third example described may be used for a third machine learning model. In at least one embodiment, any combination of tasks within the training system 1304 may be used according to the requirements of each respective machine learning model. In at least one embodiment, one or more of the machine learning models may already be trained and ready for deployment, so the machine learning model may not be processed by the training system 1304 in any way and may be implemented by the deployment system 1306.

[0159] In at least one embodiment, depending on the implementation or embodiment, the one or more output models 1316 and / or the one or more pre-trained models 1406 may include any type of machine learning model. In at least one embodiment, but not limited to, the machine learning model used by the system 1400 may include one or more machine learning models using linear regression, logistic regression, decision trees, support vector machines (SVM), naive Bayes, k-nearest neighbors (Knn), K-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoders, convolutional, recursive, perceptrons, long / short term memory (LSTM), Hopfield, Boltzmann, deep belief, deconvolution, generative adversarial, liquid state machines, etc.) and / or other types of machine learning models.

[0160] In at least one embodiment, the training pipeline 1404 may include AI-assisted annotation, as described herein at least with respect to Fig. 15BIn at least one embodiment, the labeled data 1312 (e.g., traditional annotations) may be generated by any number of techniques. In at least one embodiment, the labels or other annotations may be generated in a drawing program (e.g., an annotation program), a computer-aided design (CAD) program, a labeling program, other types of programs suitable for generating annotations or labels for ground truth, and / or may be hand-drawn in some examples. In at least one embodiment, the ground truth data may be synthetically generated (e.g., generated from a computer model or rendering), truly generated (e.g., designed and generated from real-world data), machine automated (e.g., using feature analysis and learning to extract features from the data, and then generating labels), manually annotated (e.g., a labeler or annotation expert, defining the location of the label), and / or a combination thereof. In at least one embodiment, for each instance of imaging data 1308 (or other data types used by machine learning models), there may be corresponding ground truth data generated by the training system 1304. In at least one embodiment, AI-assisted annotation may be performed as part of the deployment pipeline 1410; supplementing or replacing the AI-assisted annotation included in the training pipeline 1404. In at least one embodiment, the system 1400 may include a multi-layer platform that may include a software layer (e.g., software 1318) of a diagnostic application (or other application type) that performs one or more medical imaging and diagnostic functions. In at least one embodiment, the system 1400 may be communicatively coupled (e.g., via an encrypted link) to a PACS server network of one or more facilities. In at least one embodiment, the system 1400 may be configured to access and reference data from a PACS server to perform operations such as training a machine learning model, deploying a machine learning model, image processing, reasoning, and / or other operations.

[0161] In at least one embodiment, the software layer may be implemented as a securely encrypted and / or authenticated API through which applications or containers may be enabled (e.g., called) from one or more external environments (e.g., facility 1302). In at least one embodiment, the application may then call or execute one or more services 1320 to perform computing, AI, or visualization tasks associated with the respective application, and the software 1318 and / or services 1320 may utilize hardware 1322 to perform processing tasks in an efficient and effective manner.

[0162] In at least one embodiment, the deployment system 1306 may execute a deployment pipeline 1410. In at least one embodiment, the deployment pipeline 1410 may include any number of applications (and / or other data types) that may be applied sequentially, non-sequentially, or otherwise to imaging data generated by an imaging device, a sequencing device, a genomic device, etc., including AI-assisted annotations as described above. In at least one embodiment, as described herein, a deployment pipeline 1410 for an individual device may be referred to as a virtual instrument for the device (e.g., a virtual ultrasound instrument, a virtual CT scanning instrument, a virtual sequencing instrument, etc.). In at least one embodiment, for a single device, there may be more than one deployment pipeline 1410, depending on the information desired from the data generated by the device. In at least one embodiment, there may be a first deployment pipeline 1410 in the case where an abnormality is desired to be detected from an MRI machine, and there may be a second deployment pipeline 1410 in the case where image enhancement is desired from the output of the MRI machine.

[0163] In at least one embodiment, the image generation application may include a processing task that includes the use of a machine learning model. In at least one embodiment, a user may wish to use their own machine learning model, or select a machine learning model from the model registry 1324. In at least one embodiment, a user may implement their own machine learning model or select a machine learning model for inclusion in an application that performs a processing task. In at least one embodiment, the application may be optional and customizable, and by defining the construction of the application, the deployment and implementation of the application for a particular user is presented as a more seamless user experience. In at least one embodiment, by leveraging other features of the system 1400, such as services 1320 and hardware 1322, the deployment pipeline 1410 may be even more user-friendly, provide easier integration, and produce more accurate, efficient, and timely results.

[0164] In at least one embodiment, the deployment system 1306 may include a user interface 1414 (e.g., a graphical user interface, a web interface, etc.) that may be used to select applications to be included in one or more deployment pipelines 1410, to modify or change applications or parameters or configurations thereof, to use and interact with the deployment pipelines 1410 during setup and / or deployment, and / or to otherwise interact with the deployment system 1306. In at least one embodiment, although not shown with respect to the training system 1304, the user interface 1414 (or a different user interface) may be used to select models to be used in the deployment system 1306, to select models to be trained or retrained in the training system 1304, and / or to otherwise interact with the training system 1304.

[0165] In at least one embodiment, in addition to the application orchestration system 1428, a pipeline manager 1412 may be used to manage the interactions between the applications or containers of one or more deployment pipelines 1410 and the services 1320 and / or hardware 1322. In at least one embodiment, the pipeline manager 1412 may be configured to facilitate the interactions from application to application, from application to service 1320, and / or from application or service to hardware 1322. In at least one embodiment, although illustrated as included in the software 1318, this is not intended to be limiting, and in some examples the pipeline manager 1412 may be included in the service 1320. In at least one embodiment, the application orchestration system 1428 (e.g., Kubernetes, DOCKER, etc.) may include a container orchestration system that may group applications into multiple containers that are coordinated, managed, scaled, and deployed as logical units. In at least one embodiment, by associating applications from one or more deployment pipelines 1410 (e.g., rebuilt applications, staged applications, etc.) with individual containers, each application can be executed in a self-contained environment (e.g., at the kernel level) to improve speed and efficiency.

[0166] In at least one embodiment, each application and / or container (or image thereof) can be developed, modified, and deployed separately (e.g., a first user or developer can develop, modify, and deploy a first application and a second user or developer can develop, modify, and deploy a second application independent of the first user or developer), which can allow for focus and attention on the tasks of a single application and / or one or more containers without being hindered by the tasks of another application or one or more containers. In at least one embodiment, pipeline manager 1412 and application orchestration system 1428 can assist in communication and collaboration between different containers or applications. In at least one embodiment, application orchestration system 1428 and / or pipeline manager 1412 can facilitate communication within and between each application or container and communication of resource sharing within and between each application or container, as long as the expected input and / or output of each container or application is known by the system (e.g., based on the construction of the application or container). In at least one embodiment, because one or more applications or containers in deployment pipeline 1410 can share the same services and resources, application orchestration system 1428 can coordinate, load balance, and determine the sharing of services or resources between and among various applications or containers. In at least one embodiment, the scheduler can be used to track resource requirements of applications or containers, current or planned use of those resources, and resource availability. Thus, in at least one embodiment, the scheduler can allocate resources to different applications and allocate resources between and among applications, taking into account the needs and availability of the system. In some examples, the scheduler (and / or other components of the application orchestration system 1428) can be based on constraints imposed on the system (e.g., user constraints), such as quality of service (QoS), urgency of the need for data output (e.g., to determine whether to perform real-time processing or delayed processing), etc.

[0167] In at least one embodiment, the services 1320 utilized and shared by the applications or containers in the deployment system 1306 may include computing services 1416, AI services 1418, visualization services 1420, and / or other service types. In at least one embodiment, the application may call (e.g., execute) one or more of the services 1320 to perform processing operations for the application. In at least one embodiment, the application may utilize computing services 1416 to perform supercomputing or other high performance computing (HPC) tasks. In at least one embodiment, one or more computing services 1416 may be utilized to perform parallel processing (e.g., using a parallel computing platform 1430) to process data substantially simultaneously by one or more applications and / or one or more tasks of a single application. In at least one embodiment, a parallel computing platform 1430 (e.g., NVIDIA's CUDA) may implement general computing on a GPU (GPGPU) (e.g., GPU 1422). In at least one embodiment, the software layer of the parallel computing platform 1430 may provide access to a virtual instruction set and parallel computing elements of a GPU to execute computing kernels. In at least one embodiment, the parallel computing platform 1430 may include memory, and in some embodiments, memory may be shared between and among multiple containers, and / or between and among different processing tasks within a single container. In at least one embodiment, inter-process communication (IPC) calls may be generated for multiple containers and / or multiple processes within a container to use the same data from a shared segment of the memory of the parallel computing platform 1430 (e.g., where multiple different stages of one or more applications are processing the same information). In at least one embodiment, instead of copying data and moving the data to different locations in the memory (e.g., read / write operations), the same data in the same location of the memory may be used for any number of processing tasks (e.g., at the same time, at different times, etc.). In at least one embodiment, since the data is used as a result of processing to generate new data, this information of the new location of the data may be stored and shared between various applications. In at least one embodiment, the location of the data and the location of the updated or modified data may be part of the definition of how the payload is understood in the container.

[0168] In at least one embodiment, the AI ​​service 1418 may be utilized to perform reasoning services to execute one or more machine learning models associated with the application (e.g., a task having one or more processing tasks that execute the application). In at least one embodiment, the AI ​​service 1418 may utilize the AI ​​system 1424 to execute one or more machine learning models (e.g., a neural network such as a CNN) for segmentation, reconstruction, object detection, feature detection, classification, and / or other reasoning tasks. In at least one embodiment, one or more applications of the deployment pipeline 1410 may use one or more of the output models 1316 from the training system 1304 and / or other application models to perform reasoning on imaging data. In at least one embodiment, two or more examples of reasoning using the application orchestration system 1428 (e.g., a scheduler) may be available. In at least one embodiment, the first category may include a high priority / low latency path that can achieve a higher service level agreement, such as for performing reasoning on an urgent request in an emergency, or for use by a radiologist during a diagnostic process. In at least one embodiment, the second category may include a standard priority path that may be used for requests that may not be urgent or requests that can perform analysis at a later time. In at least one embodiment, application orchestration system 1428 may allocate resources (e.g., services 1320 and / or hardware 1322 ) based on priority paths for different reasoning tasks for AI service 1418 .

[0169] In at least one embodiment, the shared storage may be mounted to the AI ​​service 1418 within the system 1400. In at least one embodiment, the shared storage may be used as a cache (or other storage device type) and may be used to process inference requests from applications. In at least one embodiment, when an inference request is submitted, the set of API instances of the deployment system 1306 may receive the request and may select one or more instances (e.g., for best fit, for load balancing, etc.) to process the request. In at least one embodiment, in order to process the request, the request may be entered into a database, a machine learning model may be found from the model registry 1324 (if not already in the cache), and a verification step may ensure that the appropriate machine learning model is loaded into the cache (e.g., shared storage) and / or a copy of the model may be saved to the cache. In at least one embodiment, if the application is not yet running or if there are insufficient application instances, a scheduler (e.g., a scheduler of the pipeline manager 1412) may be used to start the application referenced in the request. In at least one embodiment, if an inference server has not yet been started to execute the model, the inference server may be started. Any number of inference servers may be started for each model. In at least one embodiment, in a pull model where the inference servers are clustered, the model can be cached as long as load balancing is favorable. In at least one embodiment, the inference servers can be statically loaded in the corresponding distributed servers.

[0170] In at least one embodiment, inference can be performed using an inference server running in a container. In at least one embodiment, an instance of an inference server can be associated with a model (and optionally multiple versions of the model). In at least one embodiment, if an instance of an inference server does not exist when a request to perform inference on a model is received, a new instance can be loaded. In at least one embodiment, when the inference server is started, the model can be passed to the inference server so that the same container can be used to serve different models as long as the inference server runs as different instances.

[0171] In at least one embodiment, during application execution, an inference request for a given application may be received, and a container (e.g., an instance of a hosted inference server) may be loaded (if not already loaded), and a launcher may be called. In at least one embodiment, preprocessing logic in the container may load, decode, and / or perform any additional preprocessing (e.g., using the CPU and / or GPU) on the incoming data. In at least one embodiment, once the data is prepared for inference, the container may perform inference on the data as needed. In at least one embodiment, this may include a single inference call for one image (e.g., a hand X-ray), or may require inference for hundreds of images (e.g., a chest CT). In at least one embodiment, the application may summarize the results before completion, which may include, but is not limited to, a single confidence score, pixel-level segmentation, voxel-level segmentation, generating visualizations, or generating text to summarize the findings. In at least one embodiment, different priorities may be assigned to different models or applications. For example, some models may have a real-time (TAT < 1 minute) priority, while other models may have a lower priority (e.g., TAT < 10 minutes). In at least one embodiment, model execution time may be measured from a requesting mechanism or entity and may include partner network traversal time as well as execution on an inference service.

[0172] In at least one embodiment, the request transmission between the service 1320 and the reasoning application can be hidden behind a software development kit (SDK), and a robust transmission can be provided through a queue. In at least one embodiment, the request will be placed in the queue via an API for a single application / tenant ID combination, and the SDK will pull the request from the queue and provide the request to the application. In at least one embodiment, the name of the queue can be provided in the environment where the SDK picks up the queue. In at least one embodiment, asynchronous communication through queues may be useful because it can allow any instance of the application to pick up work when available. The results can be passed back through the queue to ensure that no data is lost. In at least one embodiment, the queue can also provide the ability to segment work, because the highest priority work can enter the queue connected to most instances of the application, and the lowest priority work can enter the queue connected to a single instance, which processes tasks in the order received. In at least one embodiment, the application can run on a GPU-accelerated instance generated in the cloud 1426, and the reasoning service can perform reasoning on the GPU.

[0173] In at least one embodiment, a visualization service 1420 may be utilized to generate visualizations for viewing outputs of an application and / or one or more deployment pipelines 1410. In at least one embodiment, the visualization service 1420 may utilize a GPU 1422 to generate visualizations. In at least one embodiment, the visualization service 1420 may implement rendering effects such as ray tracing to generate higher quality visualizations. In at least one embodiment, the visualizations may include, but are not limited to, 2D image rendering, 3D volume rendering, 3D volume reconstruction, 2D tomographic slices, virtual reality displays, augmented reality displays, and the like. In at least one embodiment, a virtualized environment may be used to generate a virtual interactive display or environment (e.g., a virtual environment) for users of the system (e.g., doctors, nurses, radiologists, etc.) to interact with. In at least one embodiment, the visualization service 1420 may include an internal visualizer, movie technology, and / or other rendering or image processing capabilities or functions (e.g., ray tracing, rasterization, internal optics, etc.).

[0174] In at least one embodiment, hardware 1322 may include GPUs 1422, AI systems 1424, cloud 1426, and / or any other hardware for executing training system 1304 and / or deployment system 1306. In at least one embodiment, GPUs 1422 (e.g., NVIDIA's TESLA and / or QUADRO GPUs) may include any number of GPUs that may be used to perform processing tasks for computing services 1416, AI services 1418, visualization services 1420, other services, and / or any feature or function of processing tasks software 1318. For example, with respect to AI services 1418, GPUs 1422 may be used to perform pre-processing of imaging data (or other data types used by machine learning models), post-processing of outputs of machine learning models, and / or to perform inference (e.g., to execute machine learning models). In at least one embodiment, cloud 1426, AI systems 1424, and / or other components of system 1400 may use GPUs 1422. In at least one embodiment, cloud 1426 may include a GPU optimized platform for deep learning tasks. In at least one embodiment, AI system 1424 may use a GPU, and cloud 1426 may be executed or responsible for at least a portion of deep learning or reasoning using one or more AI systems 1424. Thus, while hardware 1322 is shown as discrete components, this is not intended to be limiting, and any component of hardware 1322 may be combined with or utilized by any other component of hardware 1322.

[0175] In at least one embodiment, AI system 1424 may include a dedicated computing system (e.g., a supercomputer or HPC) configured for reasoning, deep learning, machine learning, and / or other artificial intelligence tasks. In at least one embodiment, in addition to CPU, RAM, storage, and / or other components, features, or functions, AI system 1424 (e.g., NVIDIA's DGX) may include GPU-optimized software (e.g., a software stack) that can be executed using multiple GPUs 1422. In at least one embodiment, one or more AI systems 1424 may be implemented in a cloud 1426 (e.g., in a data center) to perform some or all of the AI-based processing tasks of system 1400.

[0176] In at least one embodiment, cloud 1426 may include a GPU-accelerated infrastructure (e.g., NVIDIA's NGC) that may provide a GPU-optimized platform for executing processing tasks of system 1400. In at least one embodiment, cloud 1426 may include one or more AI systems 1424 for executing one or more AI-based tasks of system 1400 (e.g., as a hardware abstraction and scaling platform). In at least one embodiment, cloud 1426 may be integrated with an application orchestration system 1428 that utilizes multiple GPUs to enable seamless scaling and load balancing between and among applications and services 1320. In at least one embodiment, cloud 1426 may be responsible for executing at least some of services 1320 of system 1400, including compute services 1416, AI services 1418, and / or visualization services 1420, as described herein. In at least one embodiment, cloud 1426 can perform small and large batch inference (e.g., executing NVIDIA's TENSOR RT), provide accelerated parallel computing APIs and platforms 1430 (e.g., NVIDIA's CUDA), execute application orchestration system 1428 (e.g., KUBERNETES), provide graphics rendering APIs and platforms (e.g., for ray tracing, 2D graphics, 3D graphics, and / or other rendering techniques to produce higher quality movie techniques), and / or can provide other functionality for system 1400.

[0177] Fig.15A A data flow diagram of a process 1500 for training, retraining, or updating a machine learning model according to at least one embodiment is shown. In at least one embodiment, the Fig.14The system 1400 of 1400 is used as a non-limiting example to perform the process 1500. In at least one embodiment, the process 1500 can utilize the services 1320 and / or hardware 1322 of the system 1400, as described herein. In at least one embodiment, the improved model 1512 generated by the process 1500 can be executed by the deployment system 1306 for one or more containerized applications in the deployment pipeline 1410.

[0178] In at least one embodiment, model training 1314 may include retraining or updating an initial model 1504 (e.g., a pre-trained model) using new training data (e.g., new input data, such as a customer dataset 1506 and / or new ground truth data associated with the input data). In at least one embodiment, to retrain or update the initial model 1504, the output or one or more loss layers of the initial model 1504 may be reset or deleted and / or replaced with updated or new outputs or one or more loss layers. In at least one embodiment, the initial model 1504 may have previously fine-tuned parameters (e.g., weights and / or biases) retained from prior training, so training or retraining 1314 may not take as long or require as much processing as training the model from scratch. In at least one embodiment, during model training 1314, by resetting or replacing the output or one or more loss layers of the initial model 1504, the model may be retrained or updated on a new customer dataset 1506 (e.g., Fig.13 When generating predictions on image data 1308), parameters can be updated and re-adjusted for the new dataset based on the loss calculation associated with the accuracy of the output or one or more loss layers.

[0179] In at least one embodiment, the pre-trained model 1406 may be stored in a data store or registry (e.g., Fig.13Model registry 1324 of the cloud 1426). In at least one embodiment, the pre-trained model 1406 may have been at least partially trained at one or more facilities other than the facility performing the process 1500. In at least one embodiment, in order to protect the privacy and rights of patients, subjects, or customers of different facilities, the pre-trained model 1406 may have been trained on-site using customer or patient data generated on-site. In at least one embodiment, the cloud 1426 and / or other hardware 1322 may be used to train the pre-trained model 1406, but confidential privacy-protected patient data may not be transmitted to any component of the cloud 1426, used by any component of the cloud 1426, or accessed by any component of the cloud 1426 (or other non-local hardware). In at least one embodiment, where the pre-trained model 1406 is trained using patient data from more than one facility, the pre-trained model 1406 may be trained separately for each facility before training on patient or customer data from another facility. In at least one embodiment, customer or patient data from any number of facilities may be used to train pre-trained models 1406 internally and / or externally, such as in a data center or other cloud computing infrastructure, for example where the customer or patient data has been issued privacy issues (e.g., by being relinquished, used for experimental purposes, etc.), or the customer or patient data is contained in a public dataset.

[0180] In at least one embodiment, when selecting an application to use in the deployment pipeline 1410, the user may also select a machine learning model to be used for the particular application. In at least one embodiment, the user may not have a model to use, so the user may select a pre-trained model 1406 to be used with the application. In at least one embodiment, the pre-trained model 1406 may not be optimized to generate accurate results on the customer data set 1506 of the user facility (e.g., based on patient diversity, demographics, types of medical imaging equipment used, etc.). In at least one embodiment, the pre-trained model 1406 may be updated, retrained, and / or fine-tuned for use with the corresponding facility before it is deployed into the deployment pipeline 1410 for use with one or more applications.

[0181] In at least one embodiment, a user can select a pre-trained model 1406 to be updated, retrained, and / or fine-tuned, and the pre-trained model 1406 can be referred to as an initial model 1504 for the system 1304 in the training process 1500. In at least one embodiment, a customer data set 1506 (e.g., imaging data, genomic data, sequencing data, or other data types generated by equipment in a facility) can be used to perform model training 1314 (which can include, but is not limited to, transfer learning) on ​​the initial model 1504 to generate a refined model 1512. In at least one embodiment, ground truth data corresponding to the customer data set 1506 can be generated by the training system 1304. In at least one embodiment, the training system 1304 can be, at least in part, trained by clinicians, scientists, doctors, practitioners in the facility (e.g., Fig.13 The labeled clinical data 1312) generates ground truth data.

[0182] In at least one embodiment, AI-assisted annotation 1310 can be used in some examples to generate ground truth data. In at least one embodiment, AI-assisted annotation 1310 (e.g., implemented using an AI-assisted annotation SDK) can utilize a machine learning model (e.g., a neural network) to generate suggested or predicted ground truth data for a customer data set. In at least one embodiment, a user 1510 can use the annotation tool within a user interface (graphical user interface (GUI)) on a computing device 1508.

[0183] In at least one embodiment, user 1510 may interact with the GUI via computing device 1508 to edit or fine-tune (automatic) annotations. In at least one embodiment, a polygon editing feature may be used to move vertices of a polygon to a more precise or fine-tuned position.

[0184] In at least one embodiment, once the customer dataset 1506 has associated ground truth data, the ground truth data (e.g., from AI-assisted annotations, manual labeling, etc.) can be used during model training 1314 to generate a refined model 1512. In at least one embodiment, the customer dataset 1506 can be applied to the initial model 1504 any number of times, and the ground truth data can be used to update the parameters of the initial model 1504 until an acceptable level of accuracy is achieved for the refined model 1512. In at least one embodiment, once the refined model 1512 is generated, the refined model 1512 can be deployed within one or more deployment pipelines 1410 at a facility for performing one or more processing tasks on medical imaging data.

[0185] In at least one embodiment, the refined model 1512 can be uploaded to the pre-trained models 1406 in the model registry 1324 for selection by another facility. In at least one embodiment, his process can be completed at any number of facilities, so that the refined model 1512 can be further refined any number of times on new data sets to generate a more general model.

[0186] Fig. 15B is an example illustration of a client-server architecture 1532 for enhancing an annotation tool with a pre-trained annotation model according to at least one embodiment. In at least one embodiment, an AI-assisted annotation tool 1536 can be instantiated based on the client-server architecture 1532. In at least one embodiment, the annotation tool 1536 in an imaging application can assist a radiologist, for example, in identifying organs and abnormalities. In at least one embodiment, the imaging application can include a software tool that, as a non-limiting example, helps a user 1510 identify some extreme points on a specific organ of interest in an original image 1534 (e.g., in a 3D MRI or CT scan) and receive automatic annotation results for all 2D slices of the specific organ. In at least one embodiment, the results can be stored in a data storage as training data 1538 and can be used as, for example, but not limited to, ground truth data for training. In at least one embodiment, when the computing device 1508 sends extreme points for AI-assisted annotation 1310, the deep learning model can, for example, receive the data as input and return an inferred result of segmenting an organ or abnormality. In at least one embodiment, the pre-exemplified annotation tool (e.g., Fig. 15B The AI-assisted annotation tools 1536B in the can be enhanced, for example, by making an API call (e.g., API call 1544) to a server (e.g., an annotation assistant server 1540 that can include a set of pre-trained models 1542 stored in an annotation model registry). In at least one embodiment, the annotation model registry can store pre-trained models 1542 (e.g., machine learning models, such as deep learning models) that are pre-trained to perform AI-assisted annotation for specific organs or abnormalities. These models can be further updated using the training pipeline 1404. In at least one embodiment, the pre-installed annotation tools may improve over time as new labeled clinical data 1312 is added.

[0187] Such components may be used to generate various scene graphs from one or more rule sets, which may be used to generate training data or image content representing one or more scenes of a virtual environment.

[0188] Other variations are within the spirit of the present disclosure. Thus, while the disclosed technology is susceptible to various modifications and alternative constructions, certain illustrated embodiments thereof are shown in the drawings and have been described in detail above. However, it should be understood that there is no intention to limit the disclosure to one or more specific forms disclosed, but on the contrary, it is intended to cover all modifications, alternative constructions, and equivalents that fall within the spirit and scope of the disclosure as defined by the appended claims.

[0189] Unless otherwise noted, unless otherwise noted or clearly contradicted by the context, rather than as a definition of the term, the use of the terms "a", "an" and "the" and similar references in the context of describing the disclosed embodiments (particularly in the context of the appended claims) should be interpreted as covering both the singular and the plural. The terms "comprising", "having", "including" and "containing" should be interpreted as open-ended terms (meaning "including but not limited to"). The term "connected" when unmodified refers to a physical connection and should be understood as part or all of being contained in, connected to or connected together, even if there is any intervention. Unless otherwise indicated herein, the reference to numerical ranges herein is intended only to be used as a shorthand method of referring to each individual value falling within the range separately, and each individual value is incorporated into the specification as if it were recited separately herein. Unless otherwise indicated or contradicted by the context, the use of the term "set" (e.g., "a set of items") or "subset" should be interpreted as a non-empty set including one or more members. In addition, unless otherwise indicated or contradicted by the context, the term "subset" of a corresponding set does not necessarily mean a proper subset of the corresponding set, but a subset and a corresponding set may be equal.

[0190] Unless otherwise expressly indicated or clearly contradicted by the context, joint language such as phrases of the form "at least one of A, B, and C" or "at least one of A, B, and C" is understood in the context to be generally used to refer to items, and the term etc. can be A or B or C, or any non-empty subset of the set of A and B and C. For example, in a set having three members, the conjunction phrases "at least one of A, B, and C" and "at least one of A, B, and C" refer to any of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Therefore, such joint language is generally not intended to imply that certain embodiments require the presence of at least one of A, at least one of B, and at least one of C. In addition, unless otherwise stated or contradicted by the context, the term "plurality" indicates a plural state (e.g., "plurality items" means a plurality of items). The plural is at least two items, but can be a plurality when explicitly or by the context indicates. In addition, unless otherwise stated or clear from the context, the phrase "based on" means "based at least in part on" rather than "based only on".

[0191] Unless otherwise noted herein or clearly contradictory to the environment, the operations of the processes described herein may be performed in any suitable sequence. In at least one embodiment, processes such as those described herein (or their variations and / or combinations thereof) are performed under the control of one or more computer systems configured with executable instructions, and are implemented as codes (e.g., executable instructions, one or more computer programs, or one or more applications) that are jointly executed on one or more processors by hardware or a combination thereof. In at least one embodiment, the code is stored on a computer-readable storage medium, for example, in the form of a computer program, which includes a plurality of instructions that can be executed by one or more processors. In at least one embodiment, the computer-readable storage medium is a non-transitory computer-readable storage medium that does not include a transient signal (e.g., a transient electrical or electromagnetic transmission of a propagation), but includes a non-transitory data storage circuit (e.g., a buffer, a cache, and a queue) in a transceiver of a transient signal. In at least one embodiment, code (e.g., executable code or source code) is stored on a set of one or more non-transitory computer-readable storage media (or other memory for storing executable instructions) on which executable instructions are stored, and when executed (i.e., due to being executed) by one or more processors of a computer system, the computer system performs the operations described herein. In at least one embodiment, a set of non-transitory computer-readable storage media includes a plurality of non-transitory computer-readable storage media and one or more single non-transitory storage media of the plurality of non-transitory computer-readable storage media lacking all codes, while the plurality of non-transitory computer-readable storage media collectively store all codes. In at least one embodiment, the executable instructions are executed so that different instructions are executed by different processors, for example, a non-transitory computer-readable storage medium stores instructions, and a main central processing unit ("CPU") executes some instructions, while a graphics processing unit ("GPU") executes other instructions. In at least one embodiment, different components of the computer system have separate processors, and different processors execute different subsets of instructions.

[0192] Thus, in at least one embodiment, a computer system is configured to implement one or more services that individually or collectively perform the operations of the processes described herein, and such a computer system is configured with applicable hardware and / or software capable of implementing the operations. In addition, a computer system that implements at least one embodiment of the present disclosure is a single device, and in another embodiment, is a distributed computer system that includes multiple devices that operate in different ways, so that the distributed computer system performs the operations described herein, and so that a single device does not perform all operations.

[0193] The use of any and all examples or exemplary language (e.g., "such as") provided herein is intended only to better illustrate embodiments of the present disclosure and does not limit the scope of the disclosure unless otherwise stated. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the disclosure.

[0194] All references, including publications, patent applications, and patents, cited herein are hereby incorporated by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference.

[0195] In the description and claims, the terms "coupled" and "connected" and their derivatives may be used. It should be understood that these terms may not be intended as synonyms for each other. On the contrary, in specific examples, "connected" or "coupled" may be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. "Coupled" may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other.

[0196] Unless otherwise specifically stated, it is understood that throughout the specification, terms such as "processing", "calculating", "computing", "determining", etc. refer to the actions and / or processes of a computer or computing system or similar electronic computing device to process and / or convert data represented as physical quantities (e.g., electronic) in registers and / or memories of the computing system into other data similarly represented as physical quantities in the memory, registers or other such information storage, transmission or display devices of the computing system.

[0197] In a similar manner, the term "processor" may refer to any device or part of a device that processes electronic data from registers and / or memory and converts the electronic data into other electronic data that can be stored in registers and / or memory. As a non-limiting example, a "processor" may be a CPU or a GPU. A "computing platform" may include one or more processors. As used herein, a "software" process may include, for example, software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Moreover, each process may refer to multiple processes to execute instructions continuously or intermittently, sequentially or in parallel. Because a system may embody one or more methods and a method may be considered a system, the terms "system" and "method" are used interchangeably herein.

[0198] In this document, reference may be made to obtaining, acquiring, receiving or inputting analog or digital data into a subsystem, a computer system or a computer-implemented machine. Obtaining, acquiring, receiving or inputting analog and digital data may be accomplished in a variety of ways, such as by receiving data as a parameter of a function call or a call to an application programming interface. In some embodiments, the process of obtaining, acquiring, receiving or inputting analog or digital data may be accomplished by transmitting data via a serial or parallel interface. In another embodiment, the process of obtaining, acquiring, receiving or inputting analog or digital data may be accomplished by transmitting data from a providing entity to an acquiring entity via a computer network. Reference may also be made to providing, outputting, transmitting, sending or presenting analog or digital data. In various examples, the process of providing, outputting, transmitting, sending or presenting analog or digital data may be accomplished by transmitting data as an input or output parameter of a function call, an application programming interface or an inter-process communication mechanism.

[0199] Although the above discussion sets forth example implementations of the described techniques, other architectures may be used to implement the described functionality and are intended to be within the scope of the present disclosure. In addition, although specific responsibility allocations are defined above for discussion purposes, various functions and responsibilities may be allocated and divided in different ways depending on the circumstances.

[0200] Furthermore, although the subject matter has been described in language specific to structural features and / or methodological acts, it should be understood that the subject matter claimed in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as exemplary forms of implementing the claims.

Claims

1. A computer-implemented method comprising: generating a plurality of scene structures based on the sampled rule set, the plurality of scene structures defining one or more features of one or more objects in the plurality of scenes; generating a plurality of scene graphs based on the plurality of scene structures, at least one scene graph of the plurality of scene graphs representing a scene structure of the plurality of scene structures and one or more parameter values ​​corresponding to one or more objects associated with the scene structure; rendering a plurality of images of a plurality of composite scenes using the plurality of scene graphs; as well as The training dataset is updated by at least including the plurality of images of the plurality of synthetic scenes.

2. The computer-implemented method of claim 1 , wherein: The one or more characteristics of the one or more objects include at least one of an object type or a number of objects of a particular object type.

3. The computer-implemented method of claim 1 , wherein: The plurality of images include labels of objects at least partially depicted in the plurality of images.

4. The computer-implemented method of claim 1 , wherein: A scene structure among the plurality of scene structures is generated from the sampled rule set in an unsupervised manner without data annotation.

5. The computer-implemented method of claim 1 , wherein: A scene structure among the plurality of scene structures is a hierarchical structure having nodes corresponding to the objects.

6. The computer-implemented method of claim 5, further comprising: Select a node; as well as A connected node is added to the one node based on at least one rule in the sampled rule set corresponding to an expansion of the one node.

7. The computer-implemented method of claim 6, further comprising: determining the one or more parameters of the node and the connected nodes from the sampled rule set; as well as The one or more parameters are applied to the node and the connected nodes.

8. The computer-implemented method of claim 1 , further comprising: providing the training data set to a training pipeline; as well as One or more neural networks are trained using the training data set.

9. The computer-implemented method of claim 1 , wherein: The training dataset is an enhanced existing dataset.

10. A processor, comprising: One or more circuits for: generating different scene structures including one or more objects according to the iteratively sampled rule set; Rendering an image of a scene using each of the scene structures based on one or more object parameters corresponding to one or more objects in each of the scene structures; as well as The images for the various scene structures are collected into a dataset for training one or more neural networks.

11. The processor of claim 10, wherein: The one or more circuits are also configured to: Add the dataset to the existing training dataset.

12. The processor of claim 10, wherein: A rule in the iteratively sampled rule set specifies at least one relationship between the one or more objects and a scene including the one or more objects.

13. The processor of claim 10, wherein: The one or more circuits are also configured to: Respective scene graphs are generated for the respective scene structures.

14. The processor of claim 10, wherein: The one or more circuits are also configured to: Labels are rendered for the one or more objects.

15. The processor of claim 10, wherein: The processor includes at least one of the following: A system for performing graphics rendering operations; A system for performing simulation operations; Systems for performing simulated operations to test or validate autonomous machine applications; Systems for performing deep learning operations; Systems implemented using edge devices; A system comprising one or more virtual machines VM; A system implemented at least in part in a data center; or A system implemented at least in part using cloud computing resources.

16. A system comprising: One or more processing units, the one or more processing units are used to generate a plurality of images, the plurality of images being generated in part by sampling a plurality of rules to generate a plurality of scene structures associated with a plurality of objects, and rendering the plurality of images using the plurality of scene structures and object parameters associated with the plurality of objects.

17. The system of claim 16, wherein: The one or more processing units are further configured to generate a data set comprising at least the plurality of images.

18. The system of claim 16, wherein: The one or more processing units are further configured to generate one or more masks to select one or more rules from the plurality of rules.

19. The system of claim 16, wherein: The one or more processing units are further configured to generate a plurality of scene graphs based on the plurality of scene structures, the plurality of scene structures comprising parameters assigned to objects within the plurality of scene graphs.

20. The system of claim 16, wherein: The system includes at least one of the following: A system for performing graphics rendering operations; A system for performing simulation operations; Systems for performing simulated operations to test or validate autonomous machine applications; Systems for performing deep learning operations; Systems implemented using edge devices; A system comprising one or more virtual machines VM; A system implemented at least in part in a data center; or A system implemented at least in part using cloud computing resources.

Images