Image generation using one or more neural networks

Abstract

Image generation using one or more neural networks is disclosed, specifically apparatuses, systems, and techniques for removing objects from images and performing inpainting on the regions where the objects were removed. In at least one embodiment, one or more neural networks are used to remove one or more objects from one or more images, where the one or more objects are of a similar type.

Description

TECHNICAL FIELD

[0001] At least one embodiment relates to processing resources for performing and facilitating artificial intelligence. For example, at least one embodiment relates to a processor or computing system for training a neural network in accordance with various novel techniques described herein. BACKGROUND

[0002] Image and video content is primarily captured or generated using digital image technology. A benefit of digital image or video data is that it is more easily edited or manipulated than film-based image technology. However, in many cases, editing or manipulating digital image content still requires significant manual effort to produce a result that is both visually appealing and meets intended goals. BRIEF DESCRIPTION OF DRAWINGS

[0003] Various embodiments according to the present disclosure will be described in reference to the drawings, in which:

[0004] FIG. 1A , FIG. 1B and FIG. 1C input and output images are shown in accordance with at least one embodiment;

[0005] FIG. 2 a device for generating image or video content is shown in accordance with at least one embodiment;

[0006] FIG. 3 an image editor module is shown in accordance with at least one embodiment;

[0007] FIG. 4 a process for generating an image is shown in accordance with at least one embodiment;

[0008] FIG. 5 a process for removing an object from an image is shown in accordance with at least one embodiment;

[0009] FIG. 6A inference and / or training logic is shown in accordance with at least one embodiment;

[0010] FIG. 6B inference and / or training logic is shown in accordance with at least one embodiment;

[0011] FIG. 7 an example data center system is shown in accordance with at least one embodiment;

[0012] FIG. 8 a computer system is shown in accordance with at least one embodiment;

[0013] FIG. 9 a computer system is shown in accordance with at least one embodiment;

[0014] FIG. 10 A computer system, in accordance with at least one embodiment, is shown;

[0015] FIG. 11 A computer system, in accordance with at least one embodiment, is shown;

[0016] FIG. 12A A computer system, in accordance with at least one embodiment, is shown;

[0017] FIG. 12B A computer system, in accordance with at least one embodiment, is shown;

[0018] FIG. 12C A computer system, in accordance with at least one embodiment, is shown;

[0019] FIG. 12D A computer system, in accordance with at least one embodiment, is shown;

[0020] FIG. 12E and FIG. 12F A shared programming model, in accordance with at least one embodiment, is shown;

[0021] FIG. 13 An exemplary integrated circuit and associated graphics processor, in accordance with at least one embodiment, is shown;

[0022] FIG. 14A-14B An exemplary integrated circuit and associated graphics processor, in accordance with at least one embodiment, is shown;

[0023] FIG. 15A-15B Additional exemplary graphics processor logic, in accordance with at least one embodiment, is shown;

[0024] FIG. 16 A computer system, in accordance with at least one embodiment, is shown;

[0025] FIG. 17A A parallel processor, in accordance with at least one embodiment, is shown;

[0026] FIG. 17B A partition unit, in accordance with at least one embodiment, is shown;

[0027] FIG. 17C A processing cluster, in accordance with at least one embodiment, is shown;

[0028] FIG. 17D A graphics multiprocessor, in accordance with at least one embodiment, is shown;

[0029] FIG. 18 A multi-GPU system, in accordance with at least one embodiment, is shown;

[0030] FIG. 19A graphics processor is shown in accordance with at least one embodiment;

[0031] FIG. 20 Microarchitecture of a processor is shown in accordance with at least one embodiment;

[0032] FIG. 21 A deep learning application processor is shown in accordance with at least one embodiment;

[0033] FIG. 22 An example neuron morphological processor is shown in accordance with at least one embodiment;

[0034] FIG. 23 and FIG. 24 At least portions of a graphics processor are shown in accordance with at least one embodiment;

[0035] FIG. 25 At least portions of a graphics processor core are shown in accordance with at least one embodiment;

[0036] FIG. 26A-26B At least portions of a graphics processor core are shown in accordance with at least one embodiment;

[0037] FIG. 27 A parallel processing unit (“PPU”) is shown in accordance with at least one embodiment;

[0038] FIG. 28 A general processing cluster (“GPC”) is shown in accordance with at least one embodiment;

[0039] FIG. 29 A memory partition unit of a parallel processing unit (“PPU”) is shown in accordance with at least one embodiment;

[0040] FIG. 30 A streaming multiprocessor is shown in accordance with at least one embodiment;

[0041] FIG. 31 is an example dataflow graph for an advanced compute pipeline in accordance with at least one embodiment;

[0042] FIG. 32 is a system diagram of an example system for training, adapting, instantiating, and deploying machine learning models in an advanced compute pipeline in accordance with at least one embodiment;

[0043] FIG. 33A A dataflow graph of a process for training a machine learning model is shown in accordance with at least one embodiment; and

[0044] FIG. 33B is an example illustration of a client-server architecture for augmenting annotation tools with a pre-trained annotation model in accordance with at least one embodiment. Detailed Implementation

[0045] In at least one embodiment, as shown in FIG1, a user may obtain an image 100 including a scene or other content of interest. In at least one embodiment, the image may contain or represent one or more objects from which the user wishes to remove the one or more objects. In at least one embodiment, this may include one or more objects, such as vehicles 102, 104, which may obstruct a portion of the desired view, or which the user wishes to remove from the image for other purposes. In at least one embodiment, simply identifying and removing these objects is insufficient; however, to make the image of the scene visually appealing and realistic, these areas of the source image must be replaced with image content suitable for the scene. In at least one embodiment, such a process may be referred to as inpainting, where content from the scene or background (such as color, pattern, or texture) is used to fill pixels in one or more areas or regions of the image so that these areas appear, when displayed, as if the content were part of the originally captured or generated image. In at least one embodiment, as... FIG. 1C As shown, a generated image 160 may appear, in which these objects 102 and 104 have been removed and properly repaired. This image represents the scene captured in the source image 100, but the vehicles 102 and 104 are not present.

[0046] In at least one embodiment, a system may be provided that performs deep learning-based, unsupervised or semi-supervised multi-element image and video inpainting. In at least one embodiment, this can be performed using secondary reference content. In at least one embodiment, the secondary reference content may include a reference image 130, such as... FIG. 1BAs shown, the reference image 130 enables inpainting based on automatic masks derived from one or more objects in the reference image. In at least one embodiment, an encoder such as a variational autoencoder (VAE) can be trained on reconstruction of images containing a variety of object ranges. In at least one embodiment, the encoder of this VAE can then be used to process the auxiliary image and learn its feature set. In at least one embodiment, this can include learning features of the vehicle in the reference image 130. In at least one embodiment, these features can be encoded into a latent space. In at least one embodiment, this latent space can be fed to a generative network such as a generative adversarial network (GAN) as a constraint during generation of the output image 160. In at least one embodiment, the generation by this GAN can thereby exclude one or more most prominent representations of features in the auxiliary reference image 130. In at least one embodiment, pixels of this excluded area or region can be replaced in a manner that matches or corresponds in appearance to at least a direct surrounding area, portion, or region of the image. In at least one embodiment, this process can be reversible and can inpaint content to preserve features represented in this auxiliary reference input rather than remove this content based on the training process used. In at least one embodiment, since the encoder can learn entire object classes, and multiple encoders trained on different classes can be used, such an approach can provide multi-object inpainting without the need for manual selection or masking of image content. In at least one embodiment, a single encoder trained on multiple classes of objects can be used. In at least one embodiment, implicit mask application can be provided by learning representations of arbitrary and discordant extraneous objects or patterns from the reference image 130 in an unsupervised manner, and inpainting the source image 100 to preserve or eliminate this learned representation. In at least one embodiment, such an approach can enable the system to generate an output image 160 that describes removal of all vehicles from the source image 100, even if the appearance of the vehicles in the reference image 130 does not match the appearance of the vehicles in the source image 100, whether based on model, angle, color, or other aspects. In at least one embodiment, removal and inpainting can be performed for all occurrences of all types of objects (or patterns or other visual feature sets) identified from the reference image.

[0047] In at least one embodiment, as FIG. 2image editing, manipulation, or generation can be performed locally on the client device 202. In at least one embodiment, at least a portion of this functionality can be provided by an image server 220 over at least one network 212. In at least one embodiment, image content to be edited can include image or video content, where individual video frames can be processed and treated as individual images. In at least one embodiment, this image content can relate to gaming, virtual reality (VR), or augmented reality (AR) applications. In at least one embodiment, a client device 202 can include or be comprised of a device such as a desktop computer, a notebook computer, a game console, a smartphone, a tablet computer, a VR headset, AR eyewear, a wearable computer, a digital camera, or a smart television. In at least one embodiment, an image application 224 executing on an image server 220 can initiate an editing session associated with a user of a client device 202, as can be facilitated with a session manager 226 and user data stored in a user database 236, and can result in rendering or generation of image content 234 using a rendering engine 228 or other such component, sent to the client device 202 using an appropriate transport manager 222 if this type of content or platform requires it. In at least one embodiment, a client device 202 can provide images to be processed over a network 240, and can receive back edited images or newly generated images from an image server 220. In at least one embodiment, an image application 204 on a client device 202 can include an image editor 220 with at least some capabilities, where images can be edited or generated on the client device 202 without needing to be transmitted to an image server 220. In at least one embodiment, a camera 208 can be used to capture source images or reference images, a display 206 can be used to display source images, reference images, or repaired images, and an interface for manipulating images provided by an image application 204. In at least one embodiment, content managers 222, 230 can be used to store and manage source images, reference images, or generated images. In at least one embodiment, an image application 204 on a client device 202 or an application 224 on an image server 220 can include an image editor that can perform one or more image editing, manipulation, or generation tasks. In at least one embodiment, this can include removing an object from a source image using one or more neural networks. In at least one embodiment, this can include generating a new image based on a source image, but with one or more objects no longer represented in the image content. In at least one embodiment, a generator can be trained to perform a repair to fill in a region of a source image or to generate content for a region of a source image that corresponds to a removed object, so that there is no source content in the generated image.In at least one embodiment, such repair may involve using features from regions surrounding the source image to color pixels with colors, patterns, or textures that match or correspond to those regions, so that the newly generated content appears as if it were contained within the original source image.

[0048] In at least one embodiment, the image editor 320 may include, for example: FIG. 3 The components are shown in the architecture 300. In at least one embodiment, a source image 312 may be provided as input, which will be used as the primary source of image content in the generated restored image 316. In at least one embodiment, the source image will include at least one object, pattern, or element that the user wants to remove and replace with image content that matches, blends, or corresponds to other parts of the source image 312, so that the generated output image looks as if the source image 312 was captured without the object, pattern, or element. In at least one embodiment, an auxiliary reference image 302 may be provided, which will be used as a guide or for the restoration process. In at least one embodiment, the reference image 302 will include a representation of an object type that should be removed from or edited out of the source image 312 when the restored image 312 is generated. In at least one embodiment, this may result in the removal of all objects of that object category. In at least one embodiment, the user may be able to set one or more flags or other indicators to remove one or more specified objects of that category, or not remove one or more specified objects of that category. In at least one embodiment, a user may provide multiple reference images sequentially or in parallel to remove multiple types of objects from source image 312. In at least one embodiment, "removing" objects from source image 312 means generating a version of source image 312 (e.g., a repaired image 316) that includes some image content from source image 312 but excludes image content of one or more objects of the object category specified in reference image 302, wherein a repair operation is performed on regions corresponding to those one or more objects to merge, correspond, or match with other remaining image content.

[0049] In at least one embodiment, the image editor 320 may utilize one or more neural networks to identify objects of a defined category to be removed from the source image without manually generating image masks for those individual objects. In at least one embodiment, the image editor 320 implicitly determines object regions or object masks based at least in part on the features of the objects.

[0050] In at least one embodiment, input reference image 302 is provided as an indicator of a class of object to be removed from source image 312. In at least one embodiment, an encoder trained for that object class, specifically or as a group of classes, can be used, where the encoder can encode features of that class of object into a latent space. In at least one embodiment, this latent space can be provided as input to a generative model, such as a generative adversarial network (GAN), to be used as a constraint for generating an output of a repaired image 316. In at least one embodiment, there can be multiple encoders available for encoding different classes of objects. In at least one embodiment, for each object class in a group of object classes, this can include a different trained variational autoencoder (VAE) 304. In at least one embodiment, these VAEs can be viewed as experts in a mixture based on expert methods, as can be implemented using an expert manager 322 module, device, system, component, or service. In at least one embodiment, expert manager 322 can include or consist of a gating network to determine which expert to use for a given reference image 302.

[0051] In at least one embodiment, a gating network of expert manager 322 can determine an appropriate VAE 304 to use for reference image 302. In at least one embodiment, this can include having multiple VAEs encode and recreate reference image 302, and selecting the VAE that produces the best results or most accurate recreation. In at least one embodiment, this can be implemented using a model of experts (MoE) model with a feed-forward gating or decision network that assigns experts based on image features detected by that network. In at least one embodiment, these experts can be variational autoencoders (VAEs) responsible for reconstructing that input reference image. In at least one embodiment, one or more experts are selected based on reconstruction probability. In at least one embodiment, this assignment helps to assign experts according to object-level features. In at least one embodiment, a first expert model can be specific to a first class of objects, such as vehicles, while a second expert model can be specific to a second class of objects, such as dogs or animals.

[0052] In at least one embodiment, the selected VAE 304 can include an encoder, a decoder, and a loss function that are trained or otherwise used for a corresponding class of objects. In at least one embodiment, the selected VAE 304 will determine features of reference image 302 and encode them into a latent space. In at least one embodiment, this latent space can be provided as input to a generator 310, such as a GAN, as a constraint on generating a repaired image 316. In at least one embodiment, in addition to this latent space, generator 310 can accept source image 312 as input. In at least one embodiment, generator 310 can then generate a repaired image 316 that includes image content from source image 312, but does not include any image content of objects corresponding to a class or type of objects determined from reference image 302 by selected VAE 304.

[0053] In at least one embodiment, VAE 304 provides features for a detected class of objects and encodes those features into a latent space. In at least one embodiment, VAE 304 can instead filter out objects of that class, such that only features of remaining image information are encoded into that latent space. In at least one embodiment, if generator 310 takes into account objects of a determined class of objects when generating repaired image 316, it can be penalized to encourage generator 310 to ensure that such objects are not included in a final rendered image. In at least one embodiment, this approach can remove objects from narrowly defined classes or broadly defined classes as at least partially based on content of training data utilized. In at least one embodiment, an encoder trained to recognize objects of a class of cameras can include devices such as smartphones or laptops that include cameras of a broadly defined class, and can also result in removal of these objects if a reference image includes a camera. In at least one embodiment, a classification can be as broad as electronic devices, where all objects related to electronic devices can be removed. In at least one embodiment, reference image 302 need only include an example object of a class that does not necessarily correspond to a type, size, orientation, appearance, or other aspect of any object of that class that is present in source image 312 and that is desired to be removed.

[0054] In at least one embodiment, multiple passes can be used on image editor 320. In at least one embodiment, an image can be generated and an observation model such as a convolutional neural network (CNN) 318 can analyze the image and label all objects detected in the image. In at least one embodiment, the CNN can also analyze a corresponding reference image. In at least one embodiment, if an object of the same class is identified in the reference image and the generated image, then the GAN that generated the image can be informed, thereby penalizing the GAN and causing it to generate a new image. In at least one embodiment, this process can continue until no object of the class is detected in the repaired image generated by the GAN. In at least one embodiment, a discriminator of GAN 310 can include CNN 318 to label objects in generated images. In at least one embodiment, a different GAN architecture can be utilized in which the GAN itself recognizes the latent space such that it can consider these objects when generating a repaired image. In at least one embodiment, this helps to ensure that objects of a class represented in the latent space do not appear in generated images, which would otherwise result in a penalty to the generator. In at least one embodiment, an opposite approach can be taken in which a reference image includes one or more types of objects to be preserved in a generated image, and any objects not represented in an input latent space will not appear in a generated image.

[0055] In at least one embodiment, removing multiple objects from a source image can result in a repaired image that does not look realistic because important parts of the original source image can have been removed and replaced with inferred scene content. In at least one embodiment, an additional layer can be utilized in which the generated image is edited back toward the source image in an attempt to improve realism of the generated image. In at least one embodiment, if a source image has a person holding a camera and that person is removed, the camera can now appear to be floating in mid-air in the generated repaired image. In at least one embodiment, another autoencoder 320 trained on anomaly detection can be utilized. In at least one embodiment, this autoencoder 320 can be trained on images of various objects in various orientations and positions. In at least one embodiment, this autoencoder 320 can determine something is not right in the image and indicate the presence of an anomaly. In at least one embodiment, a 3D-CNN (which can replace CNN 318) can be used to localize on the anomaly. In at least one embodiment, this 3D-CNN can also be trained on anomaly detection. In at least one embodiment, given a detected anomaly, the 3D-CNN can localize the anomaly in the image and can provide this information back to GAN 310 to regenerate the image. In at least one embodiment, the CNN described initially is optional in this case, e.g., the system can find a way to regenerate the image until the GAN generated image is not anomalous. In at least one embodiment, an anomalous image can be input to a VAE which can evaluate the image generated by GAN 310 to determine the presence of an anomaly. In at least one embodiment, the GAN can then add objects or modify aspects of the image to make the image non-anomalous, e.g., in the previous example by adding a table or tripod to support the floating camera. In at least one embodiment, the final output from the anomaly correction process is an image that looks more realistic and can be more true or accurate to the human eye.

[0056] In at least one embodiment, an image encoder can not utilize a VAE for this purpose. In at least one embodiment, objects can be removed by inpainting and blending. In at least one embodiment, anomaly detection can be performed. In at least one embodiment, these steps can be performed using a 3D-CNN instead of the VAE discussed previously. In at least one embodiment, instead of using the 3D-CNN for localization, its output can be provided to a GAN and used to generate an image. In at least one embodiment, using a 3D-CNN instead of a VAE can be accompanied by corresponding changes to the generation process and loss function.

[0057] In at least one embodiment, the indicia provided to indicate the particular object to be removed depends at least in part on the type of object. In at least one embodiment, coordinates or a location indicating the object can be provided. In at least one embodiment, text can be provided that will help determine one or more objects to remove (or not remove). In at least one embodiment, if a book is to be removed from a table image, the indicia can correspond to the title or author of the book, as visible in or determinable from the image. In at least one embodiment, a color, shape, or other indicator can also be provided that can help distinguish objects of a class to be removed from those of the class that are not to be removed. In at least one embodiment, a separate model such as a CNN can be used to identify these aspects. In at least one embodiment, a CNN can locate a location corresponding to the indicia, such as a color or title, and can provide this location to assist in object removal. In at least one embodiment, a user can specify a color so that all strokes of that color are removed from a drawing image, whereupon the area of these strokes is filled with content from other locations in the image and blended with surrounding colors, strokes, or pixels.

[0058] In at least one embodiment, a user can provide a reference image by selecting from a set of images, each image showing an object of a determined class for which a trained model can be used. In at least one embodiment, a user can instead be able to perform a search for a classification using one or more keywords that can be associated with a class of objects, and can be able to select a reference image based on results of the search. In at least one embodiment, if a classification can be determined, a model can be selected that can encode features of the class of objects into a latent space without a reference image being provided.

[0059] In at least one embodiment, as FIG. 4As shown, a process 400 for removing objects from an image can be performed. In at least one embodiment, a source image and a reference image can be received 402. In at least one embodiment, the source image has one or more objects of a type or class that a user wants to remove from the image, and the reference image includes a representation of an object of that type or class. In at least one embodiment, the reference image is provided 404 to a gated network of a set of encoders. In at least one embodiment, this can include a set of variational autoencoders (VAEs), each trained on a respective class of objects. In at least one embodiment, a mixture of expert approach can be utilized to determine 406 one of the encoders that is appropriate for the class of object represented in the reference image. In at least one embodiment, the selected encoder can encode 408 features of the object class into a latent space. In at least one embodiment, the source image and the latent space can be provided 410 as inputs to a generative network such as a GAN, where the latent space is used as a constraint on image generation. In at least one embodiment, an image can be generated 412 that is based on the source image but with one or more objects of the object class removed and replaced with a scene-appropriate inpainting. In at least one embodiment, anomaly detection and / or object detection can be performed 414, and if such an anomaly or object is detected in the image generated here, another pass can be performed in which the generator is penalized and forced to regenerate another inpainted image that should be free of the anomaly or object.

[0060] In at least one embodiment, as FIG. 5 shown, a process 500 for removing objects from an image can be performed. In at least one embodiment, a first image and a second image can be received 502. In at least one embodiment, a class of objects in the second image can be determined 504, such as by using one or more encoders or other neural networks. In at least one embodiment, one or more objects of that class can be removed 506 from the first image, such as by generating a new image that is a version of the first image without the one or more objects.

[0061] Inference and training logic

[0062] FIG. 6A Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. Inference and / or training logic 615 are used to process electrical signals received from one or more sensors (e.g., microphones, cameras, and so on). For example, in some embodiments, one or more processors 610 can be used in a manner similar to that described in association with inference and / or training logic 615 as well. FIG. 6A and / or FIG. 6B Details regarding inference and / or training logic 615 are provided below in conjunction with FIGS. 6A-6C.

[0063] In at least one embodiment, inference and / or training logic 615 can include, without limitation, code and / or data storage 601 for storing forward and / or output weights and / or input / output data, and / or other parameters of neurons or layers of a neural network configured in aspects of one or more embodiments that are trained and / or used for inferencing. In at least one embodiment, training logic 615 can include or be coupled to code and / or data storage 601 for storing graph code or other software to control timing and / or order, where weight and / or other parameter information is loaded to configure logic, including integer and / or floating point units (collectively, arithmetic logic units (ALUs)). In at least one embodiment, code, such as graph code, loads weight or other parameter information into processor ALUs based on an architecture of a neural network to which that code corresponds. In at least one embodiment, code and / or data storage 601 stores weight parameters and / or input / output data of each layer of a neural network trained or used in conjunction with one or more embodiments during forward propagation of input / output data and / or weight parameters during training and / or inferencing using aspects of one or more embodiments. In at least one embodiment, any portion of code and / or data storage 601 can be included with other on-chip or off-chip data storage, including a processor’s Ll, L2, or L3 cache or system memory.

[0064] In at least one embodiment, any portion of code and / or data storage 601 can be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, code and / or data storage 601 can be cache memory, dynamic random addressable memory (“DRAM”), static random addressable memory (“SRAM”), non-volatile memory (such as flash memory), or other storage. In at least one embodiment, a choice of whether code and / or data storage 601 is internal or external to a processor, e.g., or made up of DRAM, SRAM, flash or some other storage type, can depend on available storage space on-chip versus off-chip, latency requirements of training and / or inferencing functions being performed, batch size of data used in inferencing and / or training of a neural network, or some combination of these factors.

[0065] In at least one embodiment, inference and / or training logic 615 can include, without limitation, code and / or data storage 605 for storing backward and / or output weights and / or input / output data corresponding to neurons or layers of a neural network trained and / or used for inference in aspects of one or more embodiments. In at least one embodiment, code and / or data storage 605 stores weight parameters and / or input / output data for each layer of a neural network trained or used in conjunction with one or more embodiments during backward propagation of input / output data and / or weight parameters during training and / or inference using aspects of one or more embodiments. In at least one embodiment, training logic 615 can include or be coupled to code and / or data storage 605 for storing graph code or other software to control timing and / or order, wherein weight and / or other parameter information is loaded to configure logic, including integer and / or floating point units (collectively, arithmetic logic unit(s) (ALUs)). In at least one embodiment, code, such as graph code, loads weight or other parameter information into processor ALUs based on an architecture of a neural network to which that code corresponds. In at least one embodiment, any portion of code and / or data storage 605 can be included with other on-chip or off-chip data storage, including a processor’s L1, L2, or L3 cache or system memory. In at least one embodiment, any portion of code and / or data storage 605 can be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, code and / or data storage 605 can be cache memory, DRAM, SRAM, non-volatile memory (e.g., Flash memory), or other storage. In at least one embodiment, a choice of whether code and / or data storage 605 is internal or external to a processor, e.g., whether it is made up of DRAM, SRAM, Flash memory, or some other storage type, depends on available storage on-chip versus off-chip, latency requirements of training and / or inferencing functions being performed, batch size of data used in inference and / or training of a neural network, or some combination of these factors.

[0066] In at least one embodiment, code and / or data storage 601 and code and / or data storage 605 can be separate storage structures. In at least one embodiment, code and / or data storage 601 and code and / or data storage 605 can be the same storage structure. In at least one embodiment, code and / or data storage 601 and code and / or data storage 605 can be partly the same storage structure and partly separate storage structures. In at least one embodiment, any portion of code and / or data storage 601 and code and / or data storage 605 can be included with other on-chip or off-chip data storage, including a processor’s L1, L2, or L3 cache or system memory.

[0067] In at least one embodiment, inference and / or training logic 615 can include, without limitation, one or more arithmetic logic units (“ALUs”) 610 (including integer and / or floating point units) for performing logical and / or mathematical operations based, at least in part, on training and / or inference code (e.g., graph code) or instructions therefrom. Results of such operations can result in activations (e.g., output values from layers or neurons within a neural network) stored in activation storage 620 that are functions of input / output and / or weight parameter data stored in code and / or data storage 601 and / or code and / or data storage 605. In at least one embodiment, activations stored in activation storage 620 are generated in response to execution of instructions or other code in accordance with linear algebra and / or matrix-based mathematics performed by ALUs 610, where weight values stored in code and / or data storage 605 and / or code and / or data storage 601 are used as operands along with other values such as bias values, gradient information, momentum values, or other parameters or hyperparameters, any or all of which can be stored in code and / or data storage 605 or code and / or data storage 601 or other on-chip or off-chip storage.

[0068] In at least one embodiment, one or more ALUs 610 are included in one or more processors or other hardware logic devices or circuits, while in another embodiment one or more ALUs 610 can be external to a processor or other hardware logic device or circuit using them (e.g., a co-processor). In at least one embodiment, one or more ALUs 610 can be included within execution units of a processor, or otherwise included in a bank of ALUs accessible by execution units of a processor, which can be within a same processor or distributed between different types of processors (e.g., central processing units, graphics processing units, fixed function units, etc.). In at least one embodiment, code and / or data storage 601, code and / or data storage 605, and activation storage 620 can be on a same processor or other hardware logic device or circuit, while in another embodiment they can be in different processors or other hardware logic devices or circuits, or in some combination of same and different processors or other hardware logic devices or circuits. In at least one embodiment, any portion of activation storage 620 can be included with other on-chip or off-chip data storage, including a processor’s LI, L2, or L3 cache or system memory. Moreover, inference and / or training code can be stored with other code accessible to a processor or other hardware logic or circuit, and can be fetched and / or processed using fetch, decode, schedule, execute, exit, and / or other logic circuits of a processor.

[0069] In at least one embodiment, the active memory 620 may be a cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other memory. In at least one embodiment, the active memory 620 may be wholly or partially located inside or outside one or more processors or other logic circuits. In at least one embodiment, the active memory 620 may be selected to be inside or outside the processor, for example, or may include DRAM, SRAM, flash memory, or other memory types, depending on the availability of on-chip and off-chip memory, latency requirements for performing training and / or inference functions, batch size of data used in inference and / or training neural networks, or some combination of these factors. In at least one embodiment, FIG. 6A The inference and / or training logic 615 shown can be used in conjunction with an application-specific integrated circuit (“ASIC”), such as those from Google. Processing unit, from Graphcore TM The inference processing unit (IPU) or from Intel. (e.g., "Lake Crest") processor. In at least one embodiment, FIG. 6A The inference and / or training logic 615 shown can be used in conjunction with central processing unit (“CPU”) hardware, graphics processing unit (“GPU”) hardware or other hardware (such as field programmable gate array (“FPGA”)).

[0070] FIG. 6B Inference and / or training logic 615 according to at least one embodiment is illustrated. In at least one embodiment, the inference and / or training logic 615 may include, but is not limited to, hardware logic, wherein computational resources are dedicated or otherwise exclusively used in conjunction with weight values ​​or other information corresponding to one or more layers of neurons within a neural network. In at least one embodiment, FIG. 6B The inference and / or training logic 615 shown can be used in conjunction with an application-specific integrated circuit (ASIC), such as those from Google. Processing unit, from Graphcore TM The inference processing unit (IPU) or from Intel. (e.g., "Lake Crest") processor. In at least one embodiment, FIG. 6BThe inference and / or training logic 615, as shown in FIG. 13, can be used in combination with central processing unit (CPU) hardware, graphics processing unit (GPU) hardware, or other hardware, such as field programmable gate arrays (FPGAs). In at least one embodiment, the inference and / or training logic 615 includes, without limitation, code and / or data storage 601 and code and / or data storage 605, which can be used to store code (e.g., graph code), weight values, and / or other information, including bias values, gradient information, momentum values, and / or other parameter or hyperparameter information. In FIG. 6B In at least one embodiment, each of code and / or data storage 601 and code and / or data storage 605 is associated with a dedicated computing resource, such as computing hardware 602 and computing hardware 606, respectively. In at least one embodiment, each of computing hardware 602 and computing hardware 606 includes one or more ALUs that perform only mathematical functions (e.g., linear algebraic functions) on information stored in code and / or data storage 601 and code and / or data storage 605, respectively, the results of which are stored in activation storage 620.

[0071] In at least one embodiment, each of code and / or data storage 601 and 605 and corresponding computing hardware 602 and 606, respectively, corresponds to a different layer of a neural network, such that activations resulting from one “storage / computing pair 601 / 602” of code and / or data storage 601 and computing hardware 602 are provided as input to the next “storage / computing pair 605 / 606” of code and / or data storage 605 and computing hardware 606 in order to reflect the conceptual organization of a neural network. In at least one embodiment, each storage / computing pair 601 / 602 and 605 / 606 can correspond to more than one neural network layer. In at least one embodiment, additional storage / computing pairs (not shown) can be included in inference and / or training logic 615 after or in parallel with storage / computing pairs 601 / 602 and 605 / 606.

[0072] Data Center

[0073] FIG. 7 An example data center 700 that can use at least one embodiment is shown. In at least one embodiment, data center 700 includes a data center infrastructure layer 710, a framework layer 720, a software layer 730, and an application layer 740.

[0074] In at least one embodiment, as FIG. 7As shown, the data center infrastructure layer 710 can include a resource orchestrator 712, grouped computing resources 714, and node computing resources (“node C.R.s”) 716(1)-716(N), where “N” represents a positive integer. In at least one embodiment, node C.R.s 716(1)-716(N) can include, but are not limited to, any number of central processing units (“CPUs” or “processors”), including accelerators, field programmable gate arrays (FPGAs), graphics processors, etc., memory devices (e.g., dynamic random access memory), storage devices (e.g., solid state or disk drives), network input / output (“NW I / O”) devices, network switches, virtual machines (“VMs”), power modules, cooling modules, etc. In at least one embodiment, one or more node C.R.s of node C.R.s 716(1)-716(N) can be a server having one or more of the above-described computing resources.

[0075] In at least one embodiment, grouped computing resources 714 can include individual groups of node C.R.s housed within one or more racks (not shown), or housed within a number of racks (also not shown) within various geographic locations. Individual groups of node C.R.s within grouped computing resources 714 can include groups of computing, network, memory, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, several node C.R.s including CPUs or processors can be grouped within one or more racks to provide computing resources to support one or more workloads. In at least one embodiment, one or more racks can also include any number of power modules, cooling modules, and network switches, in any combination.

[0076] In at least one embodiment, resource orchestrator 712 can configure or otherwise control one or more node C.R.s 716(1)-716(N) and / or grouped computing resources 714. In at least one embodiment, resource orchestrator 712 can include a software design infrastructure (“SDI”) management entity for data center 700. In at least one embodiment, resource orchestrator can include hardware, software, or some combination thereof.

[0077] In at least one embodiment, as FIG. 7As shown, the framework layer 720 includes a job scheduler 722, a configuration manager 724, a resource manager 726, and a distributed file system 728. In at least one embodiment, the framework layer 720 can include a framework that supports software 732 of a software layer 730 and / or one or more applications 742 of an application layer 740. In at least one embodiment, software 732 or applications 742 can include web-based service software or applications, respectively, such as services or applications provided by Amazon Web Services, Google Cloud, and Microsoft Azure. In at least one embodiment, the framework layer 720 can be, but is not limited to, a type of free and open-source software web application framework such as Apache Spark™ (hereinafter “Spark”) that can utilize the distributed file system 728 for large-scale data processing (e.g., “big data”). In at least one embodiment, the job scheduler 732 can include a Spark driver to facilitate scheduling workloads supported by various layers of the data center 700. In at least one embodiment, the configuration manager 724 can be capable of configuring different layers, such as the software layer 730 and the framework layer 720 including Spark and the distributed file system 728 for supporting large-scale data processing. In at least one embodiment, the resource manager 726 can be capable of managing clustered or grouped computing resources mapped to or allocated for supporting the distributed file system 728 and the job scheduler 722. In at least one embodiment, the clustered or grouped computing resources can include grouped computing resources 714 on the data center infrastructure layer 710. In at least one embodiment, the resource manager 726 can coordinate with the resource orchestrator 712 to manage these mapped or allocated computing resources.

[0078] In at least one embodiment, software 732 included in the software layer 730 can include software used by at least a portion of the node C.R.s 716(1)-716(N), the grouped computing resources 714, and / or the distributed file system 728 of the framework layer 720. One or more types of software can include, but are not limited to, Internet web page search software, email virus scanning software, database software, and streaming video content software.

[0079] In at least one embodiment, one or more applications 742 included in application layer 740 can include one or more types of applications used by at least portions of node C.R.s 716(1)-716(N), grouped computing resources 714, and / or distributed file system 728 of framework layer 720. One or more types of applications can include, but are not limited to, any number and / or type of genomics applications, cognitive computing, and machine learning applications including training or inferencing software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), or other machine learning applications used in conjunction with one or more embodiments.

[0080] In at least one embodiment, any of configuration manager 724, resource manager 726, and resource orchestrator 712 can implement any number and type of self-modifying actions based on any amount and type of data acquired in any technically feasible fashion. In at least one embodiment, self-modifying actions can alleviate data center operators of data center 700 from making possibly poor configuration decisions and can avoid underutilized and / or poorly performing portions of a data center.

[0081] In at least one embodiment, data center 700 can include tools, services, software, or other resources to train one or more machine learning models or use one or more machine learning models to predict or infer information in accordance with one or more embodiments described herein. For example, in at least one embodiment, a machine learning model can be trained by computing weight parameters according to a neural network architecture using software and computing resources described above with respect to data center 700. In at least one embodiment, using weight parameters computed by one or more training techniques described herein, a trained machine learning model corresponding to one or more neural networks can be used to infer or predict information using resources described above with respect to data center 700.

[0082] In at least one embodiment, a data center can use CPUs, application specific integrated circuits (ASICs), GPUs, FPGAs, or other hardware to perform training and / or inference using resources described above. Furthermore, one or more software and / or hardware resources described above can be configured as a service to allow users to train or perform information inference such as image recognition, speech recognition, or other artificial intelligence services.

[0083] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. Details regarding inference and / or training logic 615 are provided below in conjunction with FIGS. 6 and 7. FIG. 6A and / or FIG. 6BDetails are provided regarding the inference and / or training logic 615. In at least one embodiment, the inference and / or training logic 615 can be... FIG. 7 Used in systems for inference or prediction operations based at least in part on weight parameters computed using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0084] Inference and / or training logic 615 is used to perform inference and / or training operations associated with one or more embodiments. In at least one embodiment, the logic may be used with components of these graphs to generate one or more images from which objects of the identified categories have been removed.

[0085] Computer System

[0086] FIG. 8 This is a block diagram illustrating an exemplary computer system according to at least one embodiment. The exemplary computer system may be a system of interconnected devices and components, a system-on-a-chip (SoC), or some combination thereof formed with a processor, which may include an execution unit to execute instructions. In at least one embodiment, according to this disclosure, such as the embodiments described herein, computer system 800 may include, but is not limited to, components such as processor 802, whose execution unit includes logic to execute algorithms for process data. In at least one embodiment, computer system 800 may include a processor, such as those available from Intel Corporation of Santa Clara, California. Processor family, Xeon™ XScale™ and / or StrongARM™ Core TM or Nervana TM A microprocessor may be used, although other systems (including PCs, engineering workstations, set-top boxes, etc.) with other microprocessors may also be used. In at least one embodiment, the computer system 800 may execute a version of the Windows operating system available from Microsoft Corporation of Redmond, Washington, although other operating systems (such as UNIX and Linux), embedded software, and / or graphical user interfaces may also be used.

[0087] Embodiments can be used in other devices such as handheld devices and embedded applications. Some examples of handheld devices include cellular phones, Internet Protocol devices, digital cameras, personal digital assistants ("PDAs"), and handheld PCs. In at least one embodiment, embedded applications can include a microcontroller, a digital signal processor ("DSP"), a system on a chip, a network computer ("NetPC"), a set-top box, a network hub, a wide area network ("WAN") switch, or any other system that can perform one or more instructions in accordance with at least one embodiment.

[0088] In at least one embodiment, computer system 800 can include, but is not limited to, processor 802, which can include, but is not limited to, one or more execution units 808 to perform machine learning model training and / or inferencing according to the techniques described herein. In at least one embodiment, computer system 800 is a single processor desktop or server system, but in another embodiment, computer system 800 can be a multiprocessor system. In at least one embodiment, processor 802 can include, but is not limited to, a complex instruction set computer ("CISC") microprocessor, a reduced instruction set computing ("RISC") microprocessor, a very long instruction word ("VLIW") microprocessor, a processor implementing a combo of instruction sets, or any other processor device, such as a digital signal processor. In at least one embodiment, processor 802 can be coupled to a processor bus 810 that can transmit data signals between processor 802 and other components in computer system 800.

[0089] In at least one embodiment, processor 802 can include, but is not limited to, level 1 ("Ll") internal cache memory ("cache") 804. In at least one embodiment, processor 802 can have a single -level internal cache or multi-level internal cache. In at least one embodiment, cache memory can reside in the processor 802's external. Other embodiments can include a combination of internal and external cache memory depending on the specific implementation and requirements. In at least one embodiment, register file 806 can store different types of data within various registers including, but not limited to, integer registers, floating point registers, status registers, and instruction pointer registers.

[0090] In at least one embodiment, execution unit 808 includes, without limitation, logic to perform integer and floating point operations, including bit fields. In at least one embodiment, processor 802 can also include microcode (“ucode”) read-only memory (“ROM”), which stores microcode for certain macro instructions. In at least one embodiment, execution unit 808 can also include logic to handle a packed instruction set 809. In at least one embodiment, by including packed instruction set 809 in an instruction set of a general- purpose processor, along with associated circuitry to execute instructions, many multimedia applications that would typically require specialized graphics or DSP circuits can be implemented using a general purpose processor 802. In one or more embodiments, by using the full width of a processor’s data bus for operations on packed data, many multimedia applications can be accelerated and executed more efficiently because they can not need to be broken down into multiple, smaller data elements that are processed on one element at a time. In at least one embodiment, execution unit 808 can also be used in microcontrollers, embedded processors, graphics devices, DSPs, and other types of logic circuits.

[0091] In at least one embodiment, execution unit 808 can also be used in microcontrollers, embedded processors, graphics devices, DSPs, and other types of logic circuits. In at least one embodiment, computer system 800 can include, without limitation, memory 820. In at least one embodiment, memory 820 can be implemented as a Dynamic Random Access Memory (“DRAM”) device, a Static Random Access Memory (“SRAM”) device, a flash memory device, or other memory device. In at least one embodiment, memory 820 can store data 821 and / or instructions 819 that are in the form of data signals that are executable by processor 802.

[0092] In at least one embodiment, a system logic chip can be coupled to processor bus 810 and memory 820. In at least one embodiment, system logic chip can include, without limitation, a memory controller hub (“MCH”) 816 and processor 802 can communicate with MCH 816 via processor bus 810. In at least one embodiment, MCH 816 can provide a high bandwidth memory path 818 to memory 820 for instruction and data storage and for storage of graphics commands, data, and textures. In at least one embodiment, MCH 816 can direct data signals between processor 802, memory 820, and other components in computer system 800, and

[0093] In at least one embodiment, computer system 800 can use system I / O interface 822, which is a proprietary hub interface bus to couple MCH 816 to I / O controller hub (“ICH”) 830. In at least one embodiment, ICH 830 can provide a direct connection to some I / O devices and can be used to reduce the load on processor bus 810. In at least one embodiment, local I / O bus can include, without limitation, a high-speed I / O bus for connecting peripherals to memory 820, chipset, and processor 802. Examples can include, without limitation, audio controller 829, firmware hub (“Flash BIOS”) 828, wireless transceiver 826, data storage 824, legacy I / O controller 823 including user input and keyboard interfaces 825, serial expansion port 827 (e.g., Universal Serial Bus (USB)), and network controller 834. Data storage 824 can include a hard disk drive, a floppy disk drive, a CD-ROM device, a flash memory device, or other mass storage device.

[0094] In at least one embodiment, FIG. 8 A system including interconnected hardware devices or “chips” is shown, while in other embodiments, FIG. 8 An exemplary system on a chip (SoC) can be shown. In at least one embodiment, FIG. 8The devices illustrated in FIG. 8 can be interconnected with proprietary interconnects, standardized interconnects (e.g., PCIe), or some combination thereof. In at least one embodiment, one or more components of computer system 800 are interconnected using a compute express link (CXL) interconnect.

[0095] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. Details regarding inference and / or training logic 615 are provided below in conjunction with FIGS. 6L and 6M. FIG. 6A and / or FIG. 6B Details regarding inference and / or training logic 615 are provided below in conjunction with FIGS. 6L and 6M. In at least one embodiment, inference and / or training logic 615 can be used in a system to infer or predict operations based, at least in part, on weight parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein. FIG. 8 Details regarding inference and / or training logic 615 are provided below in conjunction with FIGS. 6L and 6M. In at least one embodiment, inference and / or training logic 615 can be used in a system to infer or predict operations based, at least in part, on weight parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0096] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. In at least one embodiment, logic can be used with components of these figures to generate one or more images of objects having removed identified classes.

[0097] FIG. 9 is a block diagram illustrating an electronic device 900 for utilizing processor(s) 910 in accordance with at least one embodiment. In at least one embodiment, electronic device 900 can be, for example and without limitation, a laptop, a tower server, a rack server, a blade server, a laptop computer, a desktop computer, a tablet, a mobile device, a phone, an embedded computer, or any other suitable electronic device.

[0098] In at least one embodiment, system 900 can include, without limitation, processor(s) 910 communicatively coupled to any suitable number or kind of components, peripherals, modules, or devices. In at least one embodiment, processor(s) 910 are coupled using a bus or interface, such as an I2C bus, a System Management Bus (“SMBus”), a Low Pin Count (LPC) bus, a Serial Peripheral Interface (“SPI”), a High Definition Audio (“HDA”) bus, a Serial Advanced Technology Attachment (“SATA”) bus, a Universal Serial Bus (“USB”) (versions 1, 2, 3, etc.), or a Universal Asynchronous Receiver / Transmitter (“UART”) bus. In at least one embodiment, system 900 can include, without limitation, a processor(s) 910 coupled to a memory 920 and input / output (“I / O”) components 930. FIG. 9 In at least one embodiment, system 900 is illustrated as including hardware devices or “chips” interconnected via a bus, while in other embodiments, FIG. 9 In at least one embodiment, system 900 is illustrated as an exemplary system on a chip (“SoC”). In at least one embodiment, FIG. 9 The devices illustrated in FIG. 8 can be interconnected with proprietary interconnects, standardized interconnects (e.g., PCIe), or some combination thereof. In at least one embodiment,FIG. 9 One or more components are interconnected using Computational Fast Link (CXL) interconnects.

[0099] In at least one embodiment, FIG. 9 This may include a display 924, a touchscreen 925, a touchpad 930, a near-field communication unit (“NFC”) 945, a sensor hub 940, a thermal sensor 946, a fast chipset (“EC”) 935, a trusted platform module (“TPM”) 938, a BIOS / firmware / flash memory (“BIOS, FW Flash”) 922, a DSP 960, a drive 920 (e.g., a solid-state drive (“SSD”) or a hard disk drive (“HDD”)), a wireless local area network unit (“WLAN”) 950, a Bluetooth unit 952, a wireless wide area network unit (“WWAN”) 956, a global positioning system (GPS) 955, a camera (“USB 3.0 camera”) 954 (e.g., a USB 3.0 camera), and / or a low-power double data rate (“LPDDR”) memory unit (“LPDDR3”) 915 implemented in, for example, the LPDDR3 standard. These components may each be implemented in any suitable manner.

[0100] In at least one embodiment, other components may be communicatively coupled to processor 910 via the components described herein. In at least one embodiment, accelerometer 941, ambient light sensor (“ALS”) 942, compass 943, and gyroscope 944 may be communicatively coupled to sensor hub 940. In at least one embodiment, thermal sensor 939, fan 937, keyboard 936, and touchpad 930 may be communicatively coupled to EC 935. In at least one embodiment, speaker 963, earphone 964, and microphone (“mic”) 965 may be communicatively coupled to audio unit (“audio codec and Class D amplifier”) 962, which in turn may be communicatively coupled to DSP 960. In at least one embodiment, audio unit 962 may include, for example, but not limited to, audio encoder / decoder (“codec”) and Class D amplifier. In at least one embodiment, SIM card (“SIM”) 957 may be communicatively coupled to WWAN unit 956. In at least one embodiment, components such as WLAN unit 950, Bluetooth unit 952, and WWAN unit 956 can be implemented as next-generation form factor (NGFF).

[0101] Inference and / or training logic 615 is used to perform inference and / or training operations associated with one or more embodiments. This document combines... FIG. 6A and / or FIG. 6B Details are provided regarding the inference and / or training logic 615. In at least one embodiment, the inference and / or training logic 615 can be... FIG. 9for use in systems of the present disclosure, for inference or prediction operations based at least in part on weight parameters computed using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0102] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. In at least one embodiment, logic can be used in conjunction with components of these figures to generate one or more images of objects having removed identified classes.

[0103] FIG. 10 A computer system 1000 according to at least one embodiment is shown. In at least one embodiment, computer system 1000 is configured to implement various processes and methods described throughout this disclosure.

[0104] In at least one embodiment, computer system 1000 includes, without limitation, at least one central processing unit (“CPU”) 1002 that is connected to a communication bus 1010 implemented using any suitable protocol, such as PCI (“Peripheral Component Interconnect”), peripheral component interconnect express (“PCI-Express”), AGP (“Accelerated Graphics Port”), HyperTransport, or any other bus or point-to-point communication protocol. In at least one embodiment, computer system 1000 includes, without limitation, a main memory 1004 and control logic (e.g., implemented in hardware, software, or a combination thereof) and data can be stored in main memory 1004 in the form of random access memory (“RAM”).

[0105] In at least one embodiment, computer system 1000 includes, without limitation, an input device 1008, parallel processing system 1012, and display device 1006, which can be implemented using a conventional cathode ray tube (“CRT”), liquid crystal display (“LCD”), light emitting diode (“LED”), plasma display, or other suitable display technologies. In at least one embodiment, user input is received from input device 1008 such as a keyboard, mouse, touchpad, microphone, or the like. In at least one embodiment, each of the aforementioned modules can be located on a single semiconductor platform.

[0106] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. In at least one embodiment, logic can be used in conjunction with components of these figures to generate one or more images of objects having removed identified classes. FIG. 6A and / or FIG. 6BDetails regarding the inference and / or training logic 615 are provided. In at least one embodiment, inference and / or training logic 615 can be used in the system of FIG. 11 for inferencing or predicting operations associated with neural network training operations, neural network functions, and / or architecture, or weight parameters computed using neural network use cases described herein. FIG. 10

[0107] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. In at least one embodiment, logic can be used in conjunction with components of these figures to generate one or more images of objects having removed identified classes.

[0108] FIG. 11 A computer system 1100 according to at least one embodiment is shown. In at least one embodiment, computer system 1100 includes, without limitation, a computer 1110 and a USB stick 1120. In at least one embodiment, computer 1110 can include, without limitation, any number and type of processor (not shown) and memory (not shown). In at least one embodiment, computer 1110 includes, without limitation, a server, a cloud instance, a laptop computer, and a desktop computer.

[0109] In at least one embodiment, USB stick 1120 includes, without limitation, a processing unit 1130, a USB interface 1140, and USB interface logic 1150. In at least one embodiment, processing unit 1130 can be any instruction execution system, apparatus, or device capable of executing instructions. In at least one embodiment, processing unit 1130 can include, without limitation, any number and type of processing core (not shown). In at least one embodiment, processing core 1130 includes an application-specific integrated circuit (“ASIC”) optimized to perform any number and type of operations associated with machine learning. For example, in at least one embodiment, processing core 1130 is a tensor processing unit (“TPC”) optimized to perform machine learning inferencing operations. In at least one embodiment, processing core 1130 is a visual processing unit (“VPU”) optimized to perform machine vision and machine learning inferencing operations.

[0110] ​In at least one embodiment, USB interface 1140 can be any type of USB connector or USB socket. For example, in at least one embodiment, USB interface 1140 is a USB 3.0 Type-C socket for data and power. In at least one embodiment, USB interface 1140 is a USB 3.0 Type-A connector. In at least one embodiment, USB interface logic 1150 can include any amount and type of logic that enables processing unit 1130 to interface with a device (e.g., computer 1110) via USB connector 1140.

[0111] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. Details regarding inference and / or training logic 615 are provided below in conjunction with FIGS. 6L and / or 6M. FIG. 6A and / or FIG. 6B Details regarding inference and / or training logic 615 are provided below in conjunction with FIGS. 6L and / or 6M. In at least one embodiment, inference and / or training logic 615 can be used in a system that uses neural network training operations, neural network functions and / or architectures, or neural network use cases described herein to infer or predict operations based, at least in part, on weight parameters calculated using neural network training operations. FIG. 11 Details regarding inference and / or training logic 615 are provided below in conjunction with FIGS. 6L and / or 6M. In at least one embodiment, inference and / or training logic 615 can be used in a system that uses neural network training operations, neural network functions and / or architectures, or neural network use cases described herein to infer or predict operations based, at least in part, on weight parameters calculated using neural network training operations.

[0112] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. In at least one embodiment, logic can be used with components of these figures to generate one or more images of objects having removed identified classes.

[0113] FIG. 12A An exemplary architecture is shown in which a plurality of GPUs 1210-1213 are communicatively coupled to a plurality of multi-core processors 1205-1206 via high-speed links 1240-1243 (e.g., buses, point-to-point interconnects, etc.). In one embodiment, high-speed links 1240-1243 support a communication throughput of 4GB / s, 30GB / s, 80GB / s, or higher. Various interconnect protocols can be used including, but not limited to, PCIe 4.0 or 5.0 and NVLink 2.0.

[0114] Furthermore, in one embodiment, two or more of GPUs 1210-1213 are interconnected over high-speed links 1229-1230, which can be implemented using the same or different protocol / link as used for high-speed links 1240-1243. Similarly, two or more of multi-core processors 1205-1206 can be connected over a high-speed link 1228, which can be a symmetric multi-processor (SMP) bus that runs at 20 GB / s, 30 GB / s, 120 GB / s or higher. Alternatively, similar protocols / links (e.g., over common interconnect fabric) can be used to accomplish FIG. 12A all communication between the various system components shown in FIG. 12.

[0115] In one embodiment, each of multi-core processors 1205-1206 is communicatively coupled to processor memories 1201-1202 via memory interconnects 1226-1227, respectively, and each of GPUs 1210-1213 is communicatively coupled to GPU memories 1220-1223 over GPU memory interconnects 1250-1253, respectively. Memory interconnects 1226-1227 and 1250-1253 can utilize the same or different memory access technologies. By way of example and without limitation, processor memories 1201-1202 and GPU memories 1220-1223 can be volatile memories such as dynamic random access memory (DRAM) (including stacked DRAM), graphics DDR SDRAM (GDDR) (e.g., GDDR5, GDDR6), or high-bandwidth memory (HBM), and / or can be non-volatile memories such as 3D XPoint or Nano-Ram. In one embodiment, certain portions of processor memories 1201-1202 can be volatile memory while another portion can be non-volatile memory (e.g., using a two-level memory (2LM) hierarchy).

[0116] As described below, although various processors 1205-1206 and GPUs 1210-1213 can be physically coupled to particular memories 1201-1202, 1220-1223, respectively, a unified memory architecture can be implemented in which the same virtual system address space (also referred to as “effective address” space) is distributed across the various physical memories. For example, processor memories 1201-1202 can each include 64 GB of system memory address space, and GPU memories 1220-1223 can each include 32 GB of system memory address space (resulting in a total of 256 GB of addressable memory in this example).

[0117] FIG. 12BAdditional details are shown for the interconnect between the multi-core processor 1207 and the graphics acceleration module 1246, in accordance with one example embodiment. The graphics acceleration module 1246 can include one or more GPU chips integrated on a line card that is coupled with the processor 1207 via a high-speed link 1240. Alternatively, the graphics acceleration module 1246 can be integrated on the same package or chip as the processor 1207.

[0118] In at least one embodiment, the processor 1207 shown includes a number of cores 1260A-1260D, each with a translation lookaside buffer 1261A-1261D and one or more caches 1262A-1262D. In at least one embodiment, the cores 1260A-1260D can include various other components not shown, for executing instructions and processing data. The caches 1262A-1262D can include level one (LI) and level two (L2) caches. In addition, one or more shared caches 1256 can be included in the caches 1262A-1262D and shared by the groups of cores 1260A-1260D. For example, one embodiment of the processor 1207 includes 24 cores, each with its own LI cache, twelve shared L2 caches, and twelve shared L3 caches. In that embodiment, two adjacent cores share one or more L2 and L3 caches. The processor 1207 and graphics acceleration module 1246 are connected with system memory 1214, which can include processor memory 1201-1202 in FIG. 12A

[0119] Consistency for data and instructions stored in the various caches 1262A-1262D, 1256 and system memory 1214 is maintained via inter-core communication over the coherence bus 1264. In at least one embodiment, for example, each cache can have cache coherency logic / circuitry associated with it to communicate over the coherence bus 1264 in response to detecting a read or write to a particular cache line. In one implementation, a cache snoop protocol is implemented over the coherence bus 1264 to snoop cache accesses.

[0120] ​In at least one embodiment, agent circuitry 1225 communicatively couples graphics acceleration module 1246 to coherence bus 1264, allowing graphics acceleration module 1246 to participate in a cache coherence protocol as a peer to core complex 1260A-1260D. In particular, in at least one embodiment, interface 1235 provides connectivity from a processor bus 1240 (e.g., a PCI bus, a NVLink, etc.) to agent circuitry 1225, and interface 1237 connects graphics acceleration module 1246 to link 1240.

[0121] In one implementation, accelerator integration circuit 1236 provides cache management, memory access, context management, and interrupt management services on behalf of graphics acceleration module 1246’s graphics processing engines 1231, 1232, N. In at least one embodiment, graphics processing engines 1231, 1232, N can each comprise a separate graphics processing unit (GPU). Alternatively, graphics processing engines 1231, 1232, N can comprise different types of graphics processing engines within a GPU such as graphics execution units, media processing engines (e.g., video encoders / decoders), samplers, and blit engines. In at least one embodiment, graphics acceleration module 1246 can be a GPU with a plurality of graphics processing engines 1231-1232, N or graphics processing engines 1231-1232, N can be individual GPUs integrated on a common package, line card, or chip.

[0122] In one embodiment, accelerator integration circuit 1236 includes a memory management unit (MMU) 1239 to provide memory management services for graphics processing engines 1231-1232, N. In at least one embodiment, MMU 1239 includes address translation lookaside buffers (TLBs) (not shown) for translating virtual addresses into physical addresses, address translation lookaside buffers (TLBs) (not shown) for storing cache information such as tag, dirty and invalidation bit information. In at least one embodiment, MMU 1239 translates virtual addresses into physical addresses for memory operations requested by graphics processing engines 1231-1232, N. In at least one embodiment, MMU 1239 includes memory protection mechanisms for controlling read or write access to various memory locations by graphics processing engines 1231-1232, N. In at least one embodiment, MMU 1239 includes a memory management unit (MMU) to provide different privilege levels to processes executing on graphics processing engines 1231-1232, N or to different memory locations used by processes.

[0123] In at least one embodiment, a set of registers 1245 store context data for threads executed by graphics processing engines 1231-1232, N and context management circuit 1248 manages thread contexts. For example, context management circuit 1248 can perform save and restore operations to save and restore context for individual threads during context switches (e.g., where a first thread is saved and a second thread is stored so that it can be executed by a graphics processing engine). For example, context management circuit 1248 can store current register values to a designated area in memory (e.g., identified by a context pointer) upon a context switch. Register values can then be restored when returning to a context. In one embodiment, interrupt management circuit 1247 receives and processes interrupts received from system devices.

[0124] In one implementation, MMU 1239 translates virtual / effective addresses from graphics processing engines 1231-1232, N to real / physical addresses in system memory 1214. In at least one embodiment, accelerator integration circuit 1236 supports multiple (e.g., 4, 8, 16) graphics processor modules 1246 and / or other accelerator devices. Graphics processor modules 1246 can be dedicated to a single application executing on processor 1207 or can be shared between multiple applications. In at least one embodiment, a virtualized graphics execution environment is presented in which resources of graphics processing engines 1231-1232, N are shared between multiple applications or virtual machines (VMs). In at least one embodiment, resources can be subdivided into “slices” that are assigned to different VMs and / or applications based on processing requirements and priority levels associated with VMs and / or applications.

[0125] In at least one embodiment, accelerator integration circuit 1236 performs as a bridge to the system for graphics processor 1246 and provides address translation and system memory cache services. In addition, accelerator integration circuit 1236 can provide virtualization facilities to host processor 1207 to manage virtualization of graphics processing engines 1231-1232, N, interrupts, and memory management.

[0126] Because hardware resources of graphics processing engines 1231-1232, N are explicitly mapped to real address space seen by host processor 1207, any host processor can directly address these resources using effective address values. One function of accelerator integration circuit 1236 is to physically separate graphics processing engines 1231-1232, N so that they appear as independent units to the system.

[0127] In at least one embodiment, one or more graphics memory 1233-1234, M is coupled to each graphics processing engines 1231-1232, N respectively. Graphics memory 1233-1234, M stores instructions and data for processing by each of the graphics processing engines 1231-1232, N. In at least one embodiment, graphics memory 1233-1234, M can be a volatile memory, such as DRAM (including stacked DRAM), GDDR memory (e.g., GDDR5, GDDR6), or HBM, and / or can be a non-volatile memory, such as 3D XPoint or Nano-Ram.

[0128] In one embodiment, to reduce data traffic on link 1240, bias techniques are used to ensure that data stored in graphics memory 1233-1234, M is that which is most frequently used by graphics processing engines 1231-1232, N and that which is least used by cores 1260A-1260D (at least frequently). Similarly, bias mechanisms attempt to keep data needed by cores (and preferably not graphics processing engines 1231-1232, N) in caches 1262A-1262D, 1256 and system memory 1214 of the cores.

[0129] FIG. 12C Another exemplary embodiment is shown in which accelerator integration circuit 1236 is integrated within processor 1207. In this embodiment, graphics processing engines 1231-1232, N communicate directly over high-speed link 1240 to accelerator integration circuit 1236 via interface 1237 and interface 1235 (which can also utilize any form of bus or interface protocol). Accelerator integration circuit 1236 can execute same operations as described with respect to FIG. 12B But can have higher throughput due to its close proximity to coherence bus 1264 and caches 1262A-1262D, 1256. At least one embodiment supports different programming models including a dedicated process programming model (no graphics acceleration module virtualization) and a shared programming model (with virtualization), which can include programming models controlled by accelerator integration circuit 1236 and programming models controlled by graphics acceleration module 1246.

[0130] In at least one embodiment, graphics processing engines 1231-1232, N are dedicated to a single application or process under a single operating system. In at least one embodiment, a single application can funnel other application requests to graphics processing engines 1231-1232, N, providing virtualization within a VM / partition.

[0131] In at least one embodiment, graphics processing engines 1231-1232, N can be shared by multiple VM / application partitions. In at least one embodiment, a shared model can use a hypervisor to virtualize graphics processing engines 1231-1232, N to allow access by each operating system. For a single-partition system without a hypervisor, an operating system owns graphics processing engines 1231-1232, N. In at least one embodiment, an operating system can virtualize graphics processing engines 1231-1232, N to provide access to each process or application.

[0132] In at least one embodiment, graphics acceleration module 1246 or individual graphics processing engines 1231-1232, N use a process handle to select a process element. In at least one embodiment, process elements are stored in system memory 1214 and can be addressed using effective to real address translation techniques described herein. In at least one embodiment, a process handle can be an implementation-specific value provided to a host process when it registers its context with a graphics processing engine 1231-1232, N (i.e., calls system software to add a process element to a process element linked list). In at least one embodiment, lower 16 bits of a process handle can be an offset into a process element linked list for a process element.

[0133] FIG. 12D An exemplary accelerator integration slice 1290 is shown. As used herein, a “slice” includes a specified portion of processing resources of accelerator integration circuit 1236. Application effective address space 1282 in system memory 1214 stores process elements 1283. In one embodiment, process elements 1283 are stored in response to GPU calls 1281 from applications 1280 executing on processor 1207. Process elements 1283 include process state for respective applications 1280. A work descriptor (WD) 1284 included in process element 1283 can be a single job requested by an application or can include a pointer to a queue of jobs. In at least one embodiment, WD 1284 is a pointer to a job request queue in an application’s address space 1282.

[0134] Graphics acceleration module 1246 and / or individual graphics processing engines 1231-1232, N can be shared by all or a subset of processes in a system. In at least one embodiment, can include infrastructure for setting up process state and sending WDs 1284 to graphics acceleration module 1246 to start jobs in a virtualized environment.

[0135] In at least one embodiment, a dedicated process programming model is implementation specific. In this model, a single process owns a graphics acceleration module 1246 or individual graphics processing engines 1231-1232, N. When a graphics acceleration module 1246 is owned by a single process, a hypervisor initializes accelerator integration circuit 1236 for the owned partition, and an operating system initializes accelerator integration circuit 1236 for the owned process when a graphics acceleration module 1246 is assigned.

[0136] In operation, a WD fetch unit 1291 in accelerator integration slice 1290 fetches a next WD 1284, which includes an indication of work to be completed by one or more graphics processing engines of graphics acceleration module 1246. Data from WD 1284 can be stored in registers 1245 and used by MMU 1239, interrupt management circuit 1247, and / or context management circuit 1248, as shown. For example, one embodiment of MMU 1239 includes segment / page walk circuitry to access segment / page tables 1286 within OS virtual address space 1285. Interrupt management circuit 1247 can handle interrupt events 1292 received from graphics acceleration module 1246. When performing graphics operations, effective addresses 1293 generated by graphics processing engines 1231-1232, N are translated to real addresses by MMU 1239.

[0137] In one embodiment, a same set of registers 1245 is replicated for each graphics processing engine 1231-1232, N and / or graphics acceleration module 1246, and can be initialized by a hypervisor or operating system. Each of these replicated registers can be included in accelerator integration slice 1290. Exemplary registers that can be initialized by a hypervisor are shown in Table 1.

[0138]

[0139] Exemplary registers that can be initialized by an operating system are shown in Table 2.

[0140]

[0141]

[0142] In at least one embodiment, each WD 1284 is specific to a particular graphics acceleration module 1246 and / or graphics processing engines 1231-1232, N. It includes all information needed by the graphics processing engines 1231-1232, N to complete a job, or it can be a pointer to a memory location where an application has set up a command queue of work to be completed.

[0143] FIG. 12EAdditional details are shown for one exemplary embodiment of a shared model. This embodiment includes a hypervisor real address space 1298 in which a list of process elements 1299 is stored. The hypervisor real address space 1298 is accessible via a hypervisor 1296 that virtualizes the graphics acceleration module engine for the operating system 1295.

[0144] In at least one embodiment, a shared programming model allows all processes or a subset of processes from all partitions or a subset of partitions in a system to use a graphics acceleration module 1246. There are two programming models in which a graphics acceleration module 1246 is shared by multiple processes and partitions, i.e., time-sliced sharing and graphics-directed sharing.

[0145] In at least one embodiment, in this model, the system hypervisor 1296 owns the graphics acceleration module 1246 and makes its functionality available to all operating systems 1295. For the graphics acceleration module 1246 to support virtualization by the system hypervisor 1296, the graphics acceleration module 1246 can adhere to the following requirements: 1) application job requests must be autonomous (i.e., state need not be maintained between jobs), or the graphics acceleration module 1246 must provide a context save and restore mechanism, 2) the graphics acceleration module 1246 guarantees that an application's job request completes within a specified amount of time, including any translation faults, or the graphics acceleration module 1246 provides the ability to preempt job processing, and 3) fairness between graphics acceleration module 1246 processes must be ensured when operating in a directed shared programming model.

[0146] In at least one embodiment, application 1280 is required to use a graphics acceleration module 1246 type, a work descriptor (WD), an authority mask register (AMR) value, and a context save / restore area pointer (CSRP) for an operating system 1295 system call. In at least one embodiment, the graphics acceleration module 1246 type describes a target acceleration function for the system call. In at least one embodiment, the graphics acceleration module 1246 type can be a system specific value. In at least one embodiment, the WD is formatted specifically for graphics acceleration module 1246 and can take the form of a graphics acceleration module 1246 command, a valid address pointer to a user defined structure, a valid address pointer to a command queue, or any other data structure describing work to be done by graphics acceleration module 1246. In one embodiment, the AMR value is the AMR state for the current process. In at least one embodiment, the value passed to the operating system is similar to how an application program sets the AMR. If the accelerator integration circuit 1236 and graphics acceleration module 1246 implementation does not support a user authority mask override register (UAMOR), the operating system can apply the current UAMOR value to the AMR value before passing the AMR in a hypervisor call. The hypervisor 1296 can selectively apply the current authority mask override register (AMOR) value before placing the AMR in the process element 1283. In at least one embodiment, the CSRP is one of registers 1245 that includes a valid address of a region in the application’s effective address space 1282 for graphics acceleration module 1246 to save and restore context state. This pointer is optional if state does not need to be saved between jobs or when a job is preempted. In at least one embodiment, the context save / restore region can be a fixed system memory.

[0147] Upon receiving the system call, operating system 1295 can verify that application 1280 is registered and has been granted authority to use graphics acceleration module 1246. Operating system 1295 then invokes hypervisor 1296 using the information shown in Table 3.

[0148]

[0149] Upon receiving the hypervisor call, hypervisor 1296 verifies that operating system 1295 is registered and has been granted authority to use graphics acceleration module 1246. Hypervisor 1296 then places the process element 1283 in a process element linked list of the corresponding graphics acceleration module 1246 type. The process element can include the information shown in Table 4.

[0150]

[0151]

[0152] In at least one embodiment, the management program initializes the registers 1245 of the multiple accelerator integration slices 1290.

[0153] like FIG. 12F As shown, in at least one embodiment, a unified memory is used, which is addressable via a common virtual memory address space for accessing physical processor memories 1201-1202 and GPU memories 1220-1223. In this implementation, operations performed on GPUs 1210-1213 utilize the same virtual / effective memory address space to access processor memories 1201-1202 and vice versa, thereby simplifying programmability. In one embodiment, a first portion of the virtual / effective address space is allocated to processor memory 1201, a second portion to second processor memory 1202, a third portion to GPU memory 1220, and so on. In at least one embodiment, the entire virtual / effective memory space (sometimes referred to as the effective address space) is thus distributed across each of processor memories 1201-1202 and GPU memories 1220-1223, thereby allowing any processor or GPU to access that memory using a virtual address mapped to any physical memory.

[0154] In one embodiment, the bias / coherence management circuitry 1294A-1294E within one or more MMUs 1239A-1239E ensures cache coherence between the caches of one or more host processors (e.g., 1205) and the GPUs 1210-1213, and implements biasing techniques that indicate the physical memory in which certain types of data should be stored. While in FIG. 12F Several instances of the bias / coherence management circuitry 1294A-1294E are shown, but the bias / coherence circuitry can be implemented within the MMU of one or more host processors 1205 and / or within the accelerator integrated circuit 1236.

[0155] One embodiment allows GPU-attached memory 1220-1223 to be mapped as part of system memory and accessed using shared virtual memory (SVM) techniques, but without suffering the performance penalties associated with full system cache coherency. In at least one embodiment, the ability to access GPU-attached memory 1220-1223 as system memory without the heavy cache coherency overhead provides a favorable operating environment for GPU offload. This arrangement allows software of host processor 1205 to set operands and access computation results without the overhead of traditional I / O DMA data copies. Such traditional copies include driver calls, interrupts, and memory-mapped I / O (MMIO) accesses, which are all less efficient than simple memory accesses. In at least one embodiment, the ability to access GPU-attached memory 1220-1223 without cache coherency overhead can be critical to the execution time of offloaded computations. For example, in cases with a large amount of streaming write memory traffic, cache coherency overhead can significantly reduce the effective write bandwidth seen by GPU 1210-1213. In at least one embodiment, the efficiency of operand setup, the efficiency of result access, and the efficiency of GPU computation can all play a role in determining the effectiveness of GPU offload.

[0156] In at least one embodiment, the selection of GPU bias and host processor bias is driven by a bias tracker data structure. For example, a bias table can be used, which can be a page-granularity structure (e.g., controlled at the granularity of a memory page) that includes 1 or 2 bits per GPU-attached memory page. In at least one embodiment, with or without a bias cache in GPUs 1210-1213 (e.g., to cache frequently / recently used entries of the bias table), the bias table can be implemented in the stolen memory range of one or more GPU-attached memories 1220-1223. Alternatively, the entire bias table can be maintained within the GPU.

[0157] In at least one embodiment, prior to actually accessing GPU memory, a bias table entry associated with each access to GPU-attached memory 1220-1223 is accessed, resulting in the following operations. First, local requests from GPUs 1210-1213 that find their pages in GPU bias are forwarded directly to corresponding GPU memory 1220-1223. Local requests from GPUs that find their pages in host bias are forwarded to processor 1205 (e.g., over a high-speed link as described herein). In one embodiment, requests from processor 1205 that find requested pages in host processor bias complete requests similar to normal memory reads. Alternatively, requests that point to GPU-biased pages can be forwarded to GPUs 1210-1213. In at least one embodiment, if a page is not currently in use by a GPU, the GPU can then migrate the page to host processor bias. In at least one embodiment, bias state of a page can be changed by software-based mechanisms, hardware-assisted software-based mechanisms, or in limited cases purely hardware-based mechanisms.

[0158] One mechanism for changing bias state employs an API call (e.g., OpenCL) that in turn invokes a device driver of a GPU, which in turn sends a message (or causes a command descriptor to be enqueued) to the GPU, directing the GPU to change bias state, and in certain migrations to perform a cache flush operation in the host. In at least one embodiment, the cache flush operation is used for a migration from host processor 1205 bias to GPU bias, but not for the reverse migration.

[0159] In one embodiment, cache coherency is maintained by temporarily rendering GPU-biased pages that host processor 1205 cannot cache. To access these pages, processor 1205 can request access from a GPU 1210, which can or can not grant access immediately. Thus, to reduce communication between processor 1205 and GPU 1210, it is beneficial to ensure that GPU-biased pages are pages that are needed by the GPU but not by host processor 1205, and vice versa.

[0160] Inference and / or training logic 615 are used to perform one or more embodiments. Inference and / or training logic 615 are used to determine a degree of certainty based on a probability. Determining a degree of certainty based on a probability can include identifying a threshold probability and comparing the probability to the threshold probability. If the probability is greater than or equal to the threshold probability, then a degree of certainty can be assigned to the probability. If the probability is less than the threshold probability, then a degree of uncertainty can be assigned to the probability. FIG. 6A and / or FIG. 6B Details regarding inference and / or training logic 615 are provided with respect to FIG. 6.

[0161] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. In at least one embodiment, logic can be used in conjunction with components of these figures to generate one or more images of an object having a removed identified class.

[0162] FIG. 13 Exemplary integrated circuits and associated graphics processors in accordance with various embodiments described herein are shown, which can be fabricated using one or more IP cores. In addition to the illustrated, in at least one embodiment, other logic and circuitry can be included, including additional graphics processors / cores, peripheral interface controllers or general purpose processor cores.

[0163] FIG. 13 is a block diagram illustrating an exemplary system on a chip integrated circuit 1300 that can be fabricated using one or more IP cores, in accordance with at least one embodiment. In at least one embodiment, integrated circuit 1300 includes one or more application processor(s) 1305 (e.g., CPUs), at least one graphics processor unit(s) 1310, and can additionally include an image processor 1315 and / or a video processor 1320, any of which can be a modular IP core. In at least one embodiment, integrated circuit 1300 includes peripheral or bus logic including a USB controller 1325, a UART controller 1330, an SPI / SDIO controller 1335, and an I2S / I2C controller 1340. In at least one embodiment, integrated circuit 1300 can include a display device 1345 coupled to one or more of a high-definition multimedia interface (HDMI) controller 1350 and a mobile industry processor interface (MIPI) display interface 1355. In at least one embodiment, storage can be provided by a flash memory subsystem 1360 including flash memory and a flash memory controller. Memory interface can be provided via a memory controller 1365 for access to SDRAM or SRAM memory devices. In at least one embodiment, some integrated circuits also include an embedded security engine 1370.

[0164] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. In at least one embodiment, logic can be used in conjunction with components of these figures to generate one or more images of an object having a removed identified class. FIG. 6A and / or FIG. 6B Details regarding inference and / or training logic 615 are provided herein. In at least one embodiment, inference and / or training logic 615 can be used in integrated circuit 1300 to infer or predict operations based, at least in part, on weight parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0165] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. In at least one embodiment, used in conjunction with components of these figures, logic can generate one or more images of objects with identified classes removed.

[0166] FIG. 14A-14B Exemplary integrated circuits and associated graphics processors that can be manufactured using one or more IP cores are illustrated and described. In addition to the examples provided, other logic and circuitry can be included in the exemplary integrated circuits in at least one embodiment, including additional graphics processors / cores, peripheral interface controllers, or general-purpose processor cores.

[0167] FIG. 14A-14B is a block diagram illustrating an exemplary graphics processor utilized in a SoC, in accordance with the embodiments described herein. FIG. 14A An exemplary graphics processor 1410 of a system on a chip integrated circuit that can be manufactured using one or more IP cores is illustrated, in accordance with at least one embodiment. FIG. 14B An additional exemplary graphics processor 1440 of a system on a chip integrated circuit that can be manufactured using one or more IP cores is illustrated, in accordance with at least one embodiment. FIG. 14A Graphics processor 1410 of is a low power graphics processor core. In at least one embodiment, graphics processor 1410 can be used in graphics processing unit 1310 of FIG. 13. FIG. 14B Graphics processor 1440 of is a higher performance graphics processor core. In at least one embodiment, each of graphics processors 1410, 1440 can be FIG. 13 Variations of graphics processor 1310 of FIG. 13.

[0168] In at least one embodiment, the graphics processor 1410 includes a vertex processor 1405 and one or more fragment processors 1415A-1415N (e.g., 1415A, 1415B, 1415C, 1415D to 1415N-1 and 1415N). In at least one embodiment, the graphics processor 1410 may execute different shader programs via separate logic, such that the vertex processor 1405 is optimized to perform operations for a vertex shader program, while one or more fragment processors 1415A-1415N perform fragment (e.g., pixel) shading operations for a fragment or pixel shader program. In at least one embodiment, the vertex processor 1405 performs the vertex processing stage of the 3D graphics pipeline and generates primitive and vertex data. In at least one embodiment, one or more fragment processors 1415A-1415N use the primitive and vertex data generated by the vertex processor 1405 to generate a framebuffer displayed on a display device. In at least one embodiment, the fragment processors 1415A-1415N are optimized to execute fragment shader programs as provided in the OpenGL API, which can be used to perform operations similar to those of pixel shader programs provided in the Direct 3D API.

[0169] In at least one embodiment, the graphics processor 1410 further includes one or more memory management units (MMUs) 1420A-1420B, one or more caches 1425A-1425B, and one or more circuit interconnects 1430A-1430B. In at least one embodiment, one or more MMUs 1420A-1420B provide virtual-to-physical address mapping for the graphics processor 1410 (including for vertex processors 1405 and / or fragment processors 1415A to 1415N), and may reference vertex or image / texture data stored in memory in addition to vertex or image / texture data stored in one or more caches 1425A-1425B. In at least one embodiment, one or more MMUs 1420A-1420B may be synchronized with other MMUs within the system, including with one or more application processors 1305, image processors 1315, and / or FIG. 13 The video processor 1320 is associated with one or more MMUs, enabling each processor 1305-1320 to participate in a shared or unified virtual memory system. In at least one embodiment, one or more circuit interconnects 1430A-1430B enable the graphics processor 1410 to interface with other IP cores within the SoC via the SoC's internal bus or via a direct connection.

[0170] In at least one embodiment, the graphics processor 1440 includes FIG. 14Aone or more MMUs 1420A-1420B, one or more caches 1425A-1425B, and one or more circuit interconnects 1430A-1430B of graphics processor 1410. In at least one embodiment, graphics processor 1440 includes one or more shader core(s) 1455A-1455N (e.g., 1455A, 1455B, 1455C, 1455D, 1455E, 1455F, through 1455N-1, and 1455N), which provides a unified shader core architecture in which single instruction multiple thread (SIMT) instruction processing may be done on all enabled threads. In at least one embodiment, the number of shader cores is variable. In at least one embodiment, graphics processor 1440 includes a ring interconnect 1445 used to communicate with the shader cores, memory controller, and / or the graphics processor cores, which can be of different architecture within a single graphics processor. In at least one embodiment, graphics processor 1440 includes a task manager 1450, which can be a dedicated logic to manage execution of thread groups scheduled to the graphics processor.

[0171] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. Details regarding inference and / or training logic 615 are provided below in conjunction with FIGS. 6L and / or 6M. FIG. 6A and / or FIG. 6B Details regarding inference and / or training logic 615 are provided below in conjunction with FIGS. 6L and / or 6M. In at least one embodiment, inference and / or training logic 615 can be used in graphics processing unit 1410 for inferencing or predicting operations based on weight parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein. Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. In at least one embodiment, logic can be used with components of these figures to generate one or more images having objects of identified classes removed.

[0172] FIG. 15A-15B Additional exemplary graphics processor logic in accordance with the embodiments described herein is shown. In at least one embodiment, graphics processor 1310 includes a graphics core 1500 that can be included within graphics processor 1310, and in at least one embodiment, can be a unified shader core 1455A-1455N as shown. FIG. 15A In at least one embodiment, graphics processor 1310 includes a graphics core 1500 that can be included within graphics processor 1310, and in at least one embodiment, can be a unified shader core 1455A-1455N as shown. FIG. 13 In at least one embodiment, graphics processor 1310 includes a graphics core 1500 that can be included within graphics processor 1310, and in at least one embodiment, can be a unified shader core 1455A-1455N as shown. FIG. 14B In at least one embodiment, graphics processor 1310 includes a graphics core 1500 that can be included within graphics processor 1310, and in at least one embodiment, can be a unified shader core 1455A-1455N as shown. FIG. 15B In at least one embodiment, graphics processor 1310 includes a graphics core 1500 that can be included within graphics processor 1310, and in at least one embodiment, can be a unified shader core 1455A-1455N as shown.

[0173] In at least one embodiment, graphics core 1500 includes a shared instruction cache 1502, texture unit(s) 1518, and cache / shared memory 1520, which are common to execution resources within graphics core 1500. In at least one embodiment, graphics core 1500 can include multiple slices 1501A-1501N or partitions of each core and graphics processor can include multiple instances of graphics core 1500. Slices 1501A-1501N can include support logic including a local instruction cache 1504A-1504N, thread scheduler 1506A-1506N, thread dispatcher 1508A-1508N, and a set of registers 1510A-1510N. In at least one embodiment, slices 1501A-1501N can include a set of additional functional units (AFUs 1512A-1512N), floating point units (FPUs 1514A-1514N), integer arithmetic logic units (ALUs 1516A-1516N), address computation units (ACUs 1513A-1513N), double precision floating point units (DPFPUs 1515A-1515N), and matrix processing units (MPUs 1517A-1517N).

[0174] In at least one embodiment, FPUs 1514A-1514N can perform single precision (32-bit) and half precision (16-bit) floating point operations, while DPFPUs 1515A-1515N perform double precision (64-bit) floating point operations. In at least one embodiment, ALUs 1516A-1516N can perform variable precision integer operations in 8-bit, 16-bit, and 32-bit precision, and can be configured to operate in mixed precision arithmetic. In at least one embodiment, MPUs 1517A-1517N can also be configured for mixed precision matrix operations including half precision floating point operations and 8-bit integer operations. In at least one embodiment, MPUs 1517-1517N can perform various matrix operations to accelerate machine learning application frameworks including enabling support for accelerated general matrix to matrix multiplication (GEMM). In at least one embodiment, AFUs 1512A-1512N can perform additional logical operations not supported by floating or integer units including trigonometric operations (e.g., sine, cosine, etc.).

[0175] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. Details regarding inference and / or training logic 615 are provided below in conjunction with FIGS. 6L and / or 6M. In at least one embodiment, inference and / or training logic 615 is used in conjunction with components of perception system 610, pre-processing system 620, decision system 630, motion planning system 640, and / or motion execution system 650. FIG. 6A and / or FIG. 6BDetails regarding the inference and / or training logic 615 are provided. In at least one embodiment, the inference and / or training logic 615 can be used in graphics core 1500 for inferencing or predicting operations based, at least in part, on weight parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0176] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. In at least one embodiment, logic can be used in conjunction with components of these figures to generate one or more images of objects having removed identified classes.

[0177] FIG. 15B A general purpose processing unit (GPGPU) 1530 is shown in at least one embodiment, which can be configured to enable highly parallel compute operations to be performed by a group of graphics processing units. In at least one embodiment, GPGPU 1530 can be linked directly to other instances of GPGPU 1530 to create a multi-GPU cluster to improve speed of training for deep neural networks. In at least one embodiment, GPGPU 1530 includes a host interface 1532 to enable connections with host processors. In at least one embodiment, host interface 1532 is a PCI Express interface. In at least one embodiment, host interface 1532 can be a proprietary

[0178] In at least one embodiment, GPGPU 1530 includes memory 1544A-1544B coupled with compute clusters 1536A-1536H via a set of memory controllers 1542A-1542B. In at least one embodiment, memory 1544A-1544B can include various types of memory devices including dynamic random access memory (DRAM) or graphics random access memory, such as synchronous graphics random access memory (SGRAM), including graphics double data rate (GDDR) memory.

[0179] In at least one embodiment, compute clusters 1536A-1536H each include a group of graphics cores, such as graphics core 1540A-1540N. In at least one embodiment, graphics cores 1540A-1540N are configured to perform various graphics processing tasks such as three-dimensional (3D) graphics conversion, texture mapping, rasterization, and so forth. In at least one embodiment, graphics cores 1540A-1540N implement vertex shaders or geometry shaders for 3D FIG. 15Agraphics core 1500, which can include multiple types of integer and floating point logic units that can perform compute operations across a range of precisions for a computer, including precisions suitable for machine learning computations. For example, in at least one embodiment, at least a subset of floating point units in each compute cluster 1536A-1536H can be configured to perform 16- or 32-bit floating point operations, while a different subset of floating point units can be configured to perform 64-bit floating point operations.

[0180] In at least one embodiment, multiple instances of GPGPU 1530 can be configured to function as compute clusters. In at least one embodiment, communication for synchronization and data exchange for compute clusters 1536A-1536H varies between embodiments. In at least one embodiment, multiple instances of GPGPU 1530 communicate over host interface 1532. In at least one embodiment, GPGPU 1530 includes an I / O hub 1539 that couples GPGPU 1530 with a GPU link 1540 that enables direct connection to other instances of GPGPU 1530. In at least one embodiment, GPU link 1540 is coupled to a specialized GPU-to-GPU bridge that enables communication and synchronization between multiple instances of GPGP 1530. In at least one embodiment, GPU link 1540 is coupled with a high speed interconnect to transmit and receive data to other GPGPUs or parallel processors. In at least one embodiment, multiple instances of GPGPU 1530 are located in separate data processing systems and communicate over a network device that is accessible through host interface 1532. In at least one embodiment, GPU link 1540 can be configured to enable connection to a host processor in addition to or as an alternative to host interface 1532.

[0181] In at least one embodiment, GPGPU 1530 can be configured to train neural networks. In at least one embodiment, GPGPU 1530 can be used within an inferencing platform. In at least one embodiment, where GPGPU 1530 is used for inferencing, GPGPU 1530 can include fewer compute clusters 1536A-1536H relative to when GPGPU 1530 is used to train neural networks. In at least one embodiment, memory technology associated with memory 1544A-1544B can vary between inferencing and training configurations, with higher bandwidth memory technology dedicated to training configurations. In at least one embodiment, an inferencing configuration of GPGPU 1530 can support inferencing specific instructions. For example, in at least one embodiment, an inferencing configuration can provide support for one or more 8-bit integer dot product instructions that can be used during inferencing operations for deployed neural networks.

[0182] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. Details regarding inference and / or training logic 615 are provided below in conjunction with FIGS. 6 A and / or 6B. FIG. 6A and / or FIG. 6B Details regarding inference and / or training logic 615 are provided below in conjunction with FIGS. 6 A and / or 6B. In at least one embodiment, inference and / or training logic 615 can be used in GPGPU 1530 for inferencing or predicting operations based, at least in part, on weight parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0183] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. In at least one embodiment, logic can be used with components of these figures to generate one or more images of objects having removed identified classes.

[0184] FIG. 16 A block diagram of a computer system 1600 is shown in accordance with at least one embodiment. In at least one embodiment, computer system 1600 includes a processing subsystem 1601 with one or more processors 1602 and a system memory 1604 communicating via an interconnection path 1605, which can include a memory hub 1605. In at least one embodiment, memory hub 1605 can be a separate component coupled with one or more processors 1602 via communication links 1606 to system memory 1604. In at least one embodiment, memory hub 1605 can be integrated into one or more processors 1602 or in a chipset of a computer system 1600.

[0185] In at least one embodiment, processing subsystem 1601 includes one or more parallel processors 1612 coupled to memory hub 1605 via a bus or other communication link 1613. In at least one embodiment, communication link 1613 can use any one of a number of standard bus protocols, such as PCI, or be a vendor specific communication interface or communication structure. In at least one embodiment, one or more parallel processors 1612 form a computing complex that is a parallel or vector processing system that can include a large number of processing cores and / or processing clusters such as MIC (many integrated core) processors. In at least one embodiment, one or more parallel processors 1612 form a graphics processing subsystem that can output pixels to one of one or more display devices 1610A coupled via I / O Hub 1607. In at least one embodiment, one or more parallel processors 1612 can also include a display controller and display interface (not shown) to enable a direct connection to one or more display devices 1610B.

[0186] In at least one embodiment, system storage 1614 can connect to I / O Hub 1607 to provide storage mechanisms to platform 1600. In at least one embodiment, I / O

[0187] In at least one embodiment, platform 1600 can include other components not explicitly shown, such as USB or other port connections, optical storage drives, video capture devices, etc., which can also be connected to I / O Hub 1607. In at least one embodiment, interconnection of the various components of platform 1600 can be implemented using any suitable protocols, such as PCI-based protocols (e.g., PCI Express), or other bus or point-to-point communication interfaces and / or protocols, such as NV-Link high-speed interconnect, or interconnect protocols. FIG. 16

[0188] ​In at least one embodiment, parallel processor(s) 1612 include circuitry optimized for graphics and video processing, including video output circuitry, in at least one embodiment, parallel processor(s) 1612 include circuitry optimized for general use complementary to graphics and video processing in at least one embodiment, components of computing system 1600 can be integrated with one or more other system elements on a single integrated circuit. For example, in at least one embodiment, parallel processor(s) 1612, memory hub 1605, processor(s) 1602, and I / O hub 1607 can be integrated into a system on a chip (SoC) integrated circuit. In at least one embodiment, components of computing system 1600 can be integrated into a single package to form a system in a package (SIP) configuration. In at least one embodiment, components of computing system 1600 can be integrated into a multi-chip module (MCM), which can be interconnected with other multi-chip modules into a modular computing system.

[0189] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. Details regarding inference and / or training logic 615 are provided below in conjunction with FIGS. 6L and / or 6M. FIG. 6A and / or FIG. 6B Details regarding inference and / or training logic 615 are provided below in conjunction with FIGS. 6L and / or 6M.

[0190] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. In at least one embodiment, logic can be used in conjunction with components of these figures to generate one or more images of objects having removed identified classes.

[0191] Processor

[0192] FIG. 17A A parallel processor 1700, according to at least one embodiment, is shown in FIG. 17. In at least one embodiment, various components of parallel processor 1700 can be implemented using one or more integrated circuits, which can be programmable integrated circuits, application-specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). FIG. 16 Variations of one or more parallel processor(s) 1612 are shown.

[0193] In at least one embodiment, parallel processor 1700 includes a parallel processing unit 1702. In at least one embodiment, parallel processing unit 1702 includes an I / O unit 1704 that enables communication with other devices, including other instances of parallel processing unit 1702. In at least one embodiment, I / O unit 1704 can be directly connected to other devices. In at least one embodiment, I / O unit 1704 connects with other devices via use of a hub or switch interface, such as memory hub 1605. In at least one embodiment, connections between memory hub 1605 and I / O unit 1704 form a communication link 1613. In at least one embodiment, I / O unit 1704 connects with a host interface 1706 and a memory crossbar switch 1716, where host interface 1706 receives commands directed to performing processing operations and memory crossbar switch 1716 receives commands directed to performing memory operations.

[0194] In at least one embodiment, when host interface 1706 receives a command buffer via I / O unit 1704, host interface 1706 can direct work operations to execute those commands to front end 1708. In at least one embodiment, front end 1708 couples with a scheduler 1710, which is configured to distribute commands or other work items to a processing cluster array 1712. In at least one embodiment, scheduler 1710 ensures that processing cluster array 1712 is correctly configured and in an active state before tasks are assigned to processing cluster array 1712. In at least one embodiment, scheduler 1710 is implemented by firmware logic executing on a microcontroller. In at least one embodiment, microcontroller implemented scheduler 1710 is configurable to perform complex scheduling and work distribution operations, enabling fine-grained or coarse-grained thread dispatch and work allocation, and to enable quick preemption and context switching for threads executing on processing array 1712. In at least one embodiment, host software can prove a workload for scheduling on processing array 1712 through one of a number of graphics processing doorbells. In at least one embodiment, workload can then be automatically distributed by scheduler 1710 logic within microcontroller including scheduler 1710 on processing array 1712.

[0195] In at least one embodiment, processing cluster array 1712 can include up to “N” processing clusters (e.g., cluster 1714A, cluster 1714B, through cluster 1714N). In at least one embodiment, each cluster 1714A-1714N of processing cluster array 1712 can execute a large number of concurrent threads. In at least one embodiment, scheduler 1710 can allocate work to clusters 1714A-1714N of processing cluster array 1712 using various scheduling and / or work distribution algorithms, which can be determined by workload arising from each program or computational

[0196] In at least one embodiment, processing cluster array 1712 can be configured to perform a variety of types of parallel processing operations. In at least one embodiment, processing cluster array 1712 is configured to perform general-purpose parallel compute operations. For example, in at least one embodiment, processing cluster array 1712 can include logic to perform processing tasks comprising filtering of video and / or audio data, performing modeling operations, including physics operations, and performing data transformations.

[0197] In at least one embodiment, processing cluster array 1712 is configured to perform parallel graphics processing operations. In at least one embodiment, processing cluster array 1712 can include additional logic to support the performance of such graphics processing operations including, but not limited to, texture sampling logic to perform texture operations, tessellation logic, and other vertex processing logic. In at least one embodiment, processing cluster array 1712 can be configured to execute shader programs associated with the performance of graphics processing, such as but not limited to vertex shaders, tessellation shaders, geometry shaders, and pixel shaders. In at least one embodiment, parallel processing unit 1702 can transfer data to be processed from a system memory via I / O unit 1704. In at least one embodiment, during processing, results can be stored to on-chip memory (e.g., parallel processor memory 1722) during processing, before ultimately being written to system memory.

[0198] In at least one embodiment, when parallel processing unit 1702 is used to perform graphics processing, scheduler 1710 can be configured to divide incoming workloads into tasks of approximately equal size to better enable distribution of graphics processing operations across multiple clusters 1714A-1714N of processing cluster array 1712. In at least one embodiment, portions of processing cluster array 1712 can be configured to perform different types of processing. For example, in at least one embodiment, a first portion can be configured to

[0199] In at least one embodiment, processing cluster array 1712 can receive processing tasks to be executed from scheduler 1710, which receives commands defining the processing tasks from front end 1708. In at least one embodiment, processing tasks can comprise indices of data to be processed, e.g., surface (patch) data, primitive data, vertex data, and / or pixel data, as well as state parameters and commands (e.g., what programs to execute) that control how the data is to be processed. In at least one embodiment, scheduler 1710 can be configured to fetch the indices corresponding to a task, or can receive the indices from front end 1708. In at least one embodiment, front end 1708 can be configured to ensure that processing cluster array 1712 is configured in an effective state before a workload specified by an incoming command buffer (e.g., a batch- buffer, a push buffer, etc.) is initiated.

[0200] In at least one embodiment, each of one or more instances of parallel processing unit 1702 can be coupled to a parallel processor memory 1722. In at least one embodiment, parallel processor memory 1722 can be accessed by parallel processing unit 1702, either by one or more of its processing clusters 1712, its I / O units 1704, or by a memory crossbar 1716 that is coupled to both. In at least one embodiment, memory crossbar 1716 can be used to facilitate communication between GPCs 1712 and memory 1722. In at least one embodiment, a shared L2 cache 1718 can be used to cache data stored in memory 1722. In at least one embodiment, shared L2 cache 1718 can be shared by multiple GPCs 1712 in parallel processing unit 1702. In at least one embodiment, a shared cache can be shared by multiple parallel processing units 1702. In at least one embodiment, a cache memory can be shared by a multiplicity of processing clusters 1712 within a processing cluster array 1712. In at least one embodiment, GPCs 1712 can each comprise an I / O unit 1704 that can perform load and store operations to / from a memory location and an arithmetic logic unit (ALU) that performs integer and floating-point arithmetic operations and data transfer operations including memory operations and graphics operations. In at least one embodiment, arithmetic logic unit can implement various mathematical functions such as a wide variety of elementary mathematical functions, e.g., to perform general- purpose calculations, including trigonometric functions, exponential and logarithmic functions, and

[0201] In at least one embodiment, memory units 1724A-1724N can include various types of memory devices including dynamic random access memory (DRAM) or graphics random access memory, e.g., synchronous graphics random access memory (SGRAM), including graphics double data rate (GDDR) memory. In at least one embodiment, memory units 1724A-1724N can also include 3D stacked memory including, but not limited to, high bandwidth memory (HBM). In at least one embodiment, rendering targets such as frame buffers or texture maps can be stored across memory units 1724A-1724N allowing partition units 1720A-1720N to write portions of each rendering target in parallel to effectively use available bandwidth of parallel processor memory 1722. In at least one embodiment, local instances of parallel processor memory 1722 can be excluded to facilitate a unified memory design that utilizes system memory in combination with local cache memory.

[0202] In at least one embodiment, any of clusters 1714A-1714N of processing cluster array 1712 can process data to be written into any of memory locations 1724A-1724N within parallel processor memory 1722. In at least one embodiment, memory crossbar 1716 can be configured to transmit outputs of each cluster 1714A-1714N to any partition unit 1720A-1720N or another cluster 1714A-1714N, which can perform additional processing operations on the outputs. In at least one embodiment, each cluster 1714A-1714N can communicate with memory interface 1718 through memory crossbar 1716 to read from or write to various external memory devices. In at least one embodiment, memory crossbar 1716 has a connection to memory interface 1718 to communicate with I / O unit 1704, and a local instance of connection to parallel processor memory 1722, to enable processing clusters 1714A-1714N within different processing clusters 1714A-1714N to communicate with system memory or other memories not local to the parallel processing units 1702. In at least one embodiment, memory crossbar 1716 can use virtual channels to separate traffic streams between clusters 1714A-1714N and partition units 1720A-1720N.

[0203] In at least one embodiment, multiple instances of parallel processing unit 1702 can be provided on a single add-in card, or multiple add-in cards can be interconnected. In at least one embodiment, different instances of parallel processing unit 1702 can be configured to operate together as a single parallel processing unit 1702, even if the different instances have different numbers of processing cores, different amounts of local parallel processor memory, and / or other configuration differences.

[0204] FIG. 17B is a block diagram of a partition unit 1720 in accordance with at least one embodiment. In at least one embodiment, partition unit 1720 is a FIG. 17AThis is an example of one of the partitioning units 1720A-1720N. In at least one embodiment, the partitioning unit 1720 includes an L2 cache 1721, a frame buffer interface 1725, and a raster operation unit (“ROP”) 1726. The L2 cache 1721 is a read / write cache configured to perform load and store operations received from the memory crossbar switch 1716 and the ROP 1726. In at least one embodiment, the L2 cache 1721 outputs read misses and urgent write-back requests to the frame buffer interface 1725 for processing. In at least one embodiment, updates can also be sent to the frame buffer for processing via the frame buffer interface 1725. In at least one embodiment, the frame buffer interface 1725 communicates with memory cells in the parallel processor memory (such as…) FIG. 17A It interacts with one of the memory cells 1724A-1724N (e.g., within the parallel processor memory 1722).

[0205] In at least one embodiment, ROP 1726 is a processing unit that performs raster operations such as stenciling, z-testing, blending, etc. In at least one embodiment, ROP 1726 then outputs processed graphics data stored in graphics memory. In at least one embodiment, ROP 1726 includes compression logic to compress depth or color data written to memory and decompress depth or color data read from memory. In at least one embodiment, the compression logic may be lossless compression logic utilizing one or more of a variety of compression algorithms. The compression logic performed by ROP 1726 may vary based on the statistical characteristics of the data to be compressed. For example, in at least one embodiment, incremental color compression is performed based on depth and color data on a per-tile basis.

[0206] In at least one embodiment, ROP 1726 is included within each processing cluster (e.g., FIG. 17A Clusters 1714A-1714N are used instead of partition units 1720. In at least one embodiment, read and write requests for pixel data are made via memory crossbar switch 1716 instead of pixel fragment data transfer. In at least one embodiment, the processed graphics data can be displayed on a display device (such as...). FIG. 16 Displayed by one or more display devices 1610, routed by processor 1602 for further processing, or by... FIG. 17A One of the processing entities within the parallel processor 1700 is routed for further processing.

[0207] FIG. 17C This is a block diagram of a processing cluster 1714 within a parallel processing unit according to at least one embodiment. In at least one embodiment, the processing cluster is... FIG. 17Aone of the processing clusters 1714A-1714N. In at least one embodiment, one or more of processing clusters 1714 can be configured to execute a large number of threads in parallel, where a “thread” refers to an instance of a particular program executing on a particular set of input data. In at least one embodiment, Single Instruction Multiple Data (SIMD) instruction issue techniques are used to support parallel execution of a large number of threads without providing multiple independent instruction units each to execute a separate thread. In at least one embodiment, Single Instruction Multiple Thread (SIMT) techniques are used to support parallel execution of a large number of generally synchronized threads, using a common instruction unit configured to issue instructions to a set of processing engines within each processing cluster.

[0208] In at least one embodiment, operation of processing cluster 1714 can be controlled by a pipeline manager 1732 that allocates processing tasks to SIMT parallel processor cores. In at least one embodiment, pipeline manager 1732 receives instructions from scheduler 1710, and manages execution of those instructions via graphics multiprocessor 1734 and / or texture unit 1736. In at least one embodiment, graphics multiprocessor 1734 is an exemplary instance of a SIMT parallel processor core. However, in at least one embodiment, various types of SIMT parallel processor cores of differing architecture can be included within processing cluster 1714. In at least one embodiment, one or more instances of graphics multiprocessor 1734 can be included within a processing cluster 1714. In at least one embodiment, graphics multiprocessor 1734 can process data, and a data crossbar 1740 can be used to distribute processed data to one of a number of possible destinations, including other shader units. In at least one embodiment, pipeline manager 1732 can facilitate distribution by specifying destinations for processed data as a function of its origin. FIG. 17A

[0209] In at least one embodiment, each graphics multiprocessor 1734 within processing cluster 1714 can include an identical set of functional execution logic (e.g., arithmetic logic, load store units, etc.). In at least one embodiment, functional execution logic can be configured in a pipelined manner, where new instructions can be issued before previous instructions are complete. In at least one embodiment, functional execution logic supports a variety of operations including integer and floating point arithmetic, comparison operations, Boolean operations, shift operations, and a multitude of algebraic functions. In at least one embodiment, same functional -unit hardware can be leveraged to perform different operations using different data types. Any combination of

[0210] ​In at least one embodiment, instructions sent to the processing cluster 1714 constitute threads. In at least one embodiment, a group of threads executed across a set of parallel processing engines is a thread group. In at least one embodiment, the thread group executes programs on different input data. In at least one embodiment, each thread within the thread group may be assigned to a different processing engine within the graphics multiprocessor 1734. In at least one embodiment, the thread group may include fewer threads than the number of processing engines within the graphics multiprocessor 1734. In at least one embodiment, when the number of threads included in the thread group is less than the number of processing engines, one or more processing engines may be idle during a loop that is processing the thread group. In at least one embodiment, the thread group may also include more threads than the number of processing engines within the graphics multiprocessor 1734. In at least one embodiment, when the thread group includes more threads than the number of processing engines within the graphics multiprocessor 1734, processing can be performed in consecutive clock cycles. In at least one embodiment, multiple thread groups can be executed simultaneously on the graphics multiprocessor 1734.

[0211] In at least one embodiment, the graphics multiprocessor 1734 includes an internal cache memory for performing load and store operations. In at least one embodiment, the graphics multiprocessor 1734 may forgo the internal cache and use a cache memory within the processing cluster 1714 (e.g., L1 cache 1748). In at least one embodiment, each graphics multiprocessor 1734 may also access partition units (e.g., FIG. 17A The L2 cache is located within partition units 1720A-1720N, which are shared among all processing clusters 1714 and can be used to transfer data between threads. In at least one embodiment, the graphics multiprocessor 1734 can also access off-chip global memory, which may include one or more of local parallel processor memory and / or system memory. In at least one embodiment, any memory outside of the parallel processing unit 1702 can be used as global memory. In at least one embodiment, the processing cluster 1714 includes multiple instances of the graphics multiprocessor 1734, which can share common instructions and data that can be stored in the L1 cache 1748.

[0212] In at least one embodiment, each processing cluster 1714 may include a memory management unit (“MMU”) 1745 configured to map virtual addresses to physical addresses. In at least one embodiment, one or more instances of the MMU 1745 may reside in FIG. 17AIn at least one embodiment, MMU 1745 includes a set of page table entries (PTEs) used to map virtual addresses into physical addresses for task computation and optionally into cache line indices for tasks. In at least one embodiment, MMU 1745 can include an address translation lookaside buffer (TLB) or can reside within graphics multiprocessor 1734 or Ll cache or processing cluster 1714. In at least one embodiment, processing physical addresses enables data access locality to be determined for efficient request interleaving between partition units.

[0213] In at least one embodiment, processing cluster 1714 can be configured such that each graphics multiprocessor 1734 is coupled to a texture unit 1736 for performing texture mapping operations, e.g., determining texture sample positions, reading texture data, and filtering texture data. In at least one embodiment, texture data is read from an internal texture Ll cache (not shown) or from an Ll cache within graphics multiprocessor 1734 as needed, and texture data is fetched from an L2 cache, local parallel processor memory, or system memory, as needed. In at least one embodiment, each graphics multiprocessor 1734 outputs processed tasks to data crossbar 1740 in order to provide processed task data to another processing cluster 1714 for further processing or to store processed task data in an L2 cache, local parallel processor memory, or system memory via memory crossbar 1716. In at least one embodiment, preROP 1742 (pre-raster operations unit) is configured to receive data from graphics multiprocessor 1734, direct data to ROP unit, which can be located within partition unit (e.g., partition unit 1720A-1720N as described herein) or located within graphics multiprocessor 1734, in at least one embodiment. In at least one embodiment, preROP 1742 can perform optimizations to minimize or eliminate bandwidth usage, organize pixel color data, and perform address translations. FIG. 17A

[0214] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. Details regarding inference and / or training logic 615 are provided below in conjunction with FIGS. 6L and / or 6M. FIG. 6A and / or FIG. 6B Details regarding inference and / or training logic 615 are provided below in conjunction with FIGS. 6L and / or 6M.

[0215] ​Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. In at least one embodiment, logic can be used in conjunction with components of these figures to generate one or more images of objects having removed identified classes.

[0216] FIG. 17D A graphics processing unit 1734 according to at least one embodiment is shown. In at least one embodiment, graphics processing unit 1734 is coupled with a pipeline manager 1732 of processing cluster 1714. In at least one embodiment, graphics processing unit 1734 has a thread-execution pipeline that includes, without limitation, an instruction cache 1752, an instruction unit 1754, an address mapping unit 1756, a register file 1758, one or more general-purpose GPU (GPGPU) cores 1762, and one or more load / store units 1766. GPGPU cores 1762 and load / store units 1766 are coupled with a memory and cache interconnect 1768 to cache memory 1772 and shared memory 1770.

[0217] In at least one embodiment, instruction cache 1752 receives a stream of instructions to execute from pipeline manager 1732. In at least one embodiment, instructions are fetched from instruction cache 1752 and are dispatched for execution by instruction unit 1754. In one embodiment, instruction unit 1754 can dispatch instructions to be executed as groups of threads (e.g., thread blocks), with each thread in a thread block assigned to a different execution unit within GPGPU cores 1762. In at least one embodiment, instructions can access any of multiple different address spaces. In at least one embodiment, address mapping unit 1756 can be used to translate an address in a uniform address space to addresses in different memory address spaces that can be accessed by load / store units 1766.

[0218] In at least one embodiment, register file 1758 provides a set of registers for functional units of graphics processing unit 1734. In at least one embodiment, register file 1758 provides temporary storage for operands of the data paths connected to the functional units (e.g., GPGPU cores 1762, load / store units 1766) of graphics processing unit 1734. In at least one embodiment, register file 1758 is divided between each of the functional units such that there is a dedicated portion of the register file 1758 for each functional unit. In at least one embodiment, register file 1758 is split between different thread blocks being executed by graphics processing unit 1734.

[0219] In at least one embodiment, GPGPU cores 1762 can each include floating point units (FPUs) and / or integer arithmetic logic units (ALUs) that are capable of performing instructions defined by an extension of the SIMD instruction set implemented in GPGPU cores 1762. In at least one embodiment, first portions of GPGPU cores 1762 include single precision FPU’s and integer ALU’s, while second portions of GPGPU cores 1762 include double precision FPU’s. In at least one embodiment, FPU’s can implement IEEE 754-2008 standard for floating point arithmetic or enable variable precision floating point arithmetic. In at least one embodiment, GPGPU cores 1762 can additionally include one or more fixed function or special-purpose logic units to perform specific functions or

[0220] In at least one embodiment, GPGPU cores 1762 include SIMD logic capable of performing a single -instruction on multiple sets of data. In at least one embodiment, GPGPU cores 1762 can physically execute SIMD4, SIMD8, and SIMD16 instructions and logically execute SIMD1, SIMD2, and SIMD32 instructions. In at least one embodiment, the SIMD instructions for GPGPU cores are generated by a shader compiler and provided to GPGPU cores using a instruction cache or buffer. In at least one embodiment, GPGPU cores 1762 can work with a shader

[0221] In at least one embodiment, memory and cache interconnect 1768 is an interconnect network that connects each functional unit of graphics multiprocessor 1734 to register file 1758 and shared memory 1770. In at least one embodiment, memory and cache interconnect 1768 is a crossbar interconnect that allows load / store units 1766 to implement load and store operations between shared memory 1770 and register file 1758. In at least one embodiment, register file 1758 can operate at same frequency as GPGPU cores 1762, such that latency for data transfers between GPGPU cores 1762 and register file 1758 is very low. In at least one embodiment, shared memory 1770 can be used to enable communication between threads executing on functional units within graphics multiprocessor 1734. In at least one embodiment, cache memory 1772 can be used to cache texture data communicated between texture unit 1736 and functional units. In at least one embodiment, shared memory 1770 can also be used as a program managed cache. In at least one embodiment, in addition to auto cached data stored in cache memory 1772, threads executing on GPGPU cores 1762 can also store data in shared memory in a programmed manner.

[0222] In at least one embodiment, parallel processor or GPGPU as described herein is communicatively coupled to host / processor cores to accelerate graphics operations, machine learning operations, pattern analysis operations, and various general purpose GPU (GPGPU) functions. In at least one embodiment, GPU can be communicatively coupled to host processor / cores by a bus or other interconnect (e.g., a high speed

[0223] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. Details regarding inference and / or training logic 615 are provided below in conjunction with FIGS. 6L and / or 6M. In at least one embodiment, inference and / or training logic 615 is used in conjunction with components of system 600, for example, to perform inferencing operations, training FIG. 6A and / or FIG. 6BDetails regarding inference and / or training logic 615 are provided. In at least one embodiment, inference and / or training logic 615 can be used in graphics multiprocessor 1734 for inferencing or predicting operations based, at least in part, on weight parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0224] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. In at least one embodiment, logic can be used in graphics multiprocessor 1734 to generate one or more images of objects having removed identified classes.

[0225] FIG. 18 A multi-GPU computing system 1800 according to at least one embodiment is shown. In at least one embodiment, multi-GPU computing system 1800 can include a processor 1802 coupled to a plurality of general purpose graphics processing units (GPGPUs) 1806A-D via a host interface switch 1804. In at least one embodiment, host interface switch 1804 is a PCI Express switch device that couples processor 1802 to a PCI Express bus over which processor 1802 can communicate with GPGPUs 1806A-D. GPGPUs 1806A-D can be interconnected via a set of high-speed point-to-point GPU-to-GPU links 1816. In at least one embodiment, GPU-to-GPU links 1816 connect to each of GPGPUs 1806A-D via a dedicated GPU link. In at least one embodiment, P2P GPU links 1816 enable direct communication between each of GPGPUs 1806A-D without requiring communication over host interface bus 1804 to which processor 1802 is connected. In at least one embodiment, by directing GPU-to-GPU traffic over P2P GPU links 1816, host interface bus 1804 remains available for system memory accesses or communications with other instances of multi-GPU computing system 1800, e.g., via one or more network devices. While in at least one embodiment GPGPUs 1806A-D are connected to processor 1802 via host interface switch 1804, in at least one embodiment processor 1802 includes direct support for P2P GPU links 1816 and can connect directly to GPGPUs 1806A-D.

[0226] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. In at least one embodiment, logic can be used in graphics multiprocessor 1734 to generate one or more images of objects having removed identified classes. FIG. 6A and / or FIG. 6BDetails regarding the inference and / or training logic 615 are provided. In at least one embodiment, inference and / or training logic 615 can be used in multi-GPU computing system 1800 for inferencing or predicting operations based, at least in part, on weight parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0227] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. In at least one embodiment, logic can be used with components of these figures to generate one or more images of objects having removed identified classes.

[0228] FIG. 19 FIG. 19B is a block diagram of a graphics processor 1900 according to at least one embodiment. In at least one embodiment, graphics processor 1900 includes ring interconnect 1902, pipeline front-end 1904, media engine 1937, and graphics core(s) 1980A-1980N. In at least one embodiment, ring interconnect 1902 couples graphics processor 1900 to other processing units such as other graphics processors or one or more general-purpose processor cores. In at least one embodiment, graphics processor 1900 is one of a plurality of processors integrated within a multi-core processing system.

[0229] In at least one embodiment, graphics processor 1900 receives batches of commands via ring interconnect 1902. In at least one embodiment, incoming commands are interpreted by a command streamer 1903 within pipeline front-end 1904. In at least one embodiment, graphics processor 1900 includes scalable execution logic to perform 3D geometry processing and media processing via the graphics core(s) 1980A-1980N. In at least one embodiment, for 3D geometry processing commands, command streamer 1903 supplies commands to geometry pipeline 1936. In at least one embodiment, for at least some media processing commands, command streamer 1903 supplies

[0230] In at least one embodiment, graphics processor 1900 includes a scalable thread execution resource including a module core featuring 1980A-1980N (sometimes referred to as a core slice), each with multiple sub-cores 1950A-1950N, 1960A-1960N (sometimes referred to as a core sub-slice). In at least one embodiment, graphics processor 1900 can have any number of graphics cores 1980A-1980N. In at least one embodiment, graphics processor 1900 includes graphics core 1980A having at least a first sub-core 1950A and a second sub-core 1960A. In at least one embodiment, graphics processor 1900 is a low power processor with a single sub-core (e.g., 1950A). In at least one embodiment, graphics processor 1900 includes multiple graphics cores 1980A-1980N each including a set of first sub-cores 1950A-1950N and a set of second sub-cores 1960A-1960N. In at least one embodiment, each sub-core in first sub-cores 1950A-1950N includes at least a first set of execution units 1952A-1952N and media / texture samplers 1954A-1954N. In at least one embodiment, each sub-core in second sub-cores 1960A-1960N includes at least a second set of execution units 1962A-1962N and samplers 1964A-1964N. In at least one embodiment, each sub-core 1950A-1950N, 1960A-1960N shares a set of shared resources 1970A-1970N. In at least one embodiment, shared resources include shared cache memory and pixel operation logic.

[0231] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. Details regarding inference and / or training logic 615 are provided below in conjunction with FIGS. 6L and / or 6M. FIG. 6A and / or FIG. 6B Details regarding inference and / or training logic 615 are provided below in conjunction with FIGS. 6L and / or 6M. In at least one embodiment, inference and / or training logic 615 can be used in graphics processor 1900 for inferencing or predicting operations based, at least in part, on weight parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0232] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. In at least one embodiment, logic can be used in conjunction with components of these figures to generate one or more images having objects with identified classes removed.

[0233] FIG. 20is a block diagram illustrating microarchitecture for a processor 2000 according to at least one embodiment, which can include logic circuits to execute instructions. In at least one embodiment, processor 2000 can execute instructions including x86 instructions, ARM instructions, specialized instructions for application specific integrated circuits (ASICs), and the like. In at least one embodiment, processor 2000 can include registers to store packed data, such as 64-bit wide MMX TM registers enabled by Intel Corporation of Santa Clara, California in microprocessors employing MMX technology. In at least one embodiment, MMX registers available in integer and floating point form can operate with packed data elements accompanying single instruction multiple data (“SIMD”) and streaming SIMD extensions (“SSE”) instructions. In at least one embodiment, 128-bit wide XMM registers related to SSE2, SSE3, SSE4, AVX, or higher (generically referred to as “SSEx”) technology can hold such packed data operands. In at least one embodiment, processor 2000 can execute instructions to accelerate machine learning or deep learning algorithms, training, or inference.

[0234] In at least one embodiment, processor 2000 includes an in-order front-end (“front-end”) 2001 to fetch instructions to be executed and to prepare instructions for execution by other units of the processor pipeline. In at least one embodiment, front-end 2001 can include several units. In at least one embodiment, instruction prefetcher 2026 fetches instructions from memory and provides instructions to instruction decoder 2028, which in turn, decodes or interprets instructions. For example, in at least one embodiment, instruction decoder 2028 decodes a received instruction into one or more operations called “micro-instructions” or “micro-operations” (also called “micro ops” or “uops”) that the machine can execute. In at least one embodiment, instruction decoder 2028 parses instruction into an operation code that, alongside the corresponding data and control fields, can be used by micro-architecture to perform operations according to at least one embodiment. In at least one embodiment, trace cache 2030 can assemble decoded micro-instructions into program ordered sequences or traces in micro-instruction queue 2034 for execution. In at least one embodiment, when trace cache 2030 encounters a complex instruction, microcode ROM 2032 provides uops needed to complete operation.

[0235] In at least one embodiment, some instructions can be converted into a single micro- operation, while others can require several micro-operations to complete. In at least one embodiment, if more than four micro-instructions are needed to complete a single instruction, then instruction decoder 2028 can access microcode ROM 2032 to execute the instruction. In at least one embodiment, instructions can be decoded into a small number of micro-instructions to be processed at instruction decoder 2028. In at least one embodiment, if multiple micro-instructions are needed to complete an operation, then the instructions can be stored in microcode ROM 2032. In at least one embodiment, trace cache 2030 references an entry point programmable logic array (“PLA”) to determine a correct micro-instruction pointer for reading a microcode sequence from microcode ROM 2032 to complete one or more instructions, in accordance with at least one embodiment. In at least one embodiment, after microcode ROM 2032 completes sequencing of micro-operations for an instruction, front-end 2001 of a machine can resume fetching micro-operations from trace cache 2030.

[0236] In at least one embodiment, out-of-order execution engine (“out-of-order engine”) 2003 can prepare instructions for execution. In at least one embodiment, out-of-order execution logic has multiple buffers to smooth and reorder instruction flow to optimize performance as instructions are pipelined down and dispatched for execution. In at least one embodiment, out-of-order execution engine 2003 includes, without limitation, an allocator / register renamer 2040, a memory micro instruction queue 2042, an integer / float micro instruction queue 2044, a memory scheduler 2046, a fast scheduler 2002, a slow / general floating point scheduler (“slow / general FP scheduler”) 2004, and a simple floating point scheduler (“simple FP scheduler”) 2006. In at least one embodiment, fast scheduler 2002, slow / general floating point scheduler 2004, and simple floating point scheduler 2006 are also collectively referred to as “micro instruction schedulers 2002, 2004, 2006.” In at least one embodiment, allocator / register renamer 2040 allocates machine buffers and resources needed by each micro instruction to execute in sequence. In at least one embodiment, allocator / register renamer 2040 renames logical registers to entries in a register file. In at least one embodiment, allocator / register renamer 2040 also allocates entries for each micro instruction in one of two micro instruction queues, memory micro instruction queue 2042 for memory operations and integer / float micro instruction queue 2044 for non-memory operations, in front of memory scheduler 2046 and micro instruction schedulers 2002, 2004, 2006. In at least one embodiment, micro instruction schedulers 2002, 2004, 2006 determine when a micro instruction is ready to execute based on readiness of their dependent input register operand sources and availability of execution resources micro instructions need to complete. In at least one embodiment, fast scheduler 2002 of at least one embodiment can schedule on every half of a main clock cycle, while slow / general floating point scheduler 2004 and simple floating point scheduler 2006 can schedule once per main processor clock cycle. In at least one embodiment, micro instruction schedulers 2002, 2004, 2006 arbitrate for a dispatch port to dispatch a micro instruction for execution.

[0237] In at least one embodiment, execution block 2011 includes, without limitation, integer register file / bypass network 2008, floating point register file / bypass network (“FP register file / bypass network”) 2010, address generation units (“AGUs”) 2012 and 2014, fast arithmetic logic units (“fast ALUs”) 2016 and 2018, slow arithmetic logic unit (“slow ALU”) 2020, floating point ALU (“FP”) 2022, and floating point move unit (“FP move”) 2024. In at least one embodiment, integer register file / bypass network 2008 and floating point register file / bypass network 2010 are also referred to herein as “register files 2008, 2010.” In at least one embodiment, AGUs 2012 and 2014, fast ALUs 2016 and 2018, slow ALU 2020, floating point ALU 2022, and floating point move unit 2024 are also referred to herein as “execution units 2012, 2014, 2016, 2018, 2020, 2022, and 2024.” In at least one embodiment, execution block 2011 can include, without limitation, any number (including zero) and type of register files, bypass networks, address generation units, and execution units (in any combination).

[0238] In at least one embodiment, register files 2008, 2010 can be arranged between micro-instruction schedulers 2002, 2004, 2006 and execution units 2012, 2014, 2016, 2018, 2020, 2022, and 2024. In at least one embodiment, integer register file / bypass network 2008 performs integer operations. In at least one embodiment, floating point register file / bypass network 2010 performs floating point operations. In at least one embodiment, each of register files 2008, 2010 can include, without limitation, a bypass network that can bypass or forward a just-completed result that has not yet been written into a register file to a new dependee. In at least one embodiment, register files 2008, 2010 can communicate data with each other. In at least one embodiment, integer register file / bypass network 2008 can include, without limitation, two separate register files, one for low order 32 bits data, a second for high order 32 bits data. In at least one embodiment, floating point register file / bypass network 2010 can include, without limitation, 128 bit wide entries, as floating point instructions typically have operands that are 64 to 128 bits wide.

[0239] In at least one embodiment, execution units 2012, 2014, 2016, 2018, 2020, 2022, 2024 can execute instructions. In at least one embodiment, register files 2008, 2010 store integer and floating point data operand values upon which microinstructions require execution. In at least one embodiment, processor 2000 can include, without limitation, any number and combination of execution units 2012, 2014, 2016, 2018, 2020, 2022, 2024. In at least one embodiment, floating point ALU 2022 and floating point move unit 2024 can execute floating point, MMX, SIMD, AVX and SSE, or other operations, including specialized machine learning instructions. In at least one embodiment, floating point ALU 2022 can include, without limitation, a 64-bit by 64-bit floating point divider to execute divide, square root, and remainder micro-ops. In at least one embodiment, instructions for dealing with floating point values can be handled with floating point hardware. In at least one embodiment, ALU operations can be passed to fast ALUs 2016, 2018. In at least one embodiment, fast ALUs 2016, 2018 can execute fast operations with an effective latency of half a clock cycle. In at least one embodiment, most complex integer operations enter slow ALU 2020 as slow ALU 2020 can include, without limitation, integer execution hardware for long latency type of operations such as multiplies, shifts, flag logic, and branch processing. In at least one embodiment, memory load / store operations can be executed by AGUs 2012, 2014. In at least one embodiment, fast ALU 2016, fast ALU 2018, and slow ALU 2020 can execute integer operations on 64-bit data operands. In at least one embodiment, fast ALU 2016, fast ALU 2018, and slow ALU 2020 can be implemented to support a variety of data bit sizes including sixteen, thirty-two, 128, 256, etc. In at least one embodiment, floating point ALU 2022 and floating point move unit 2024 can be implemented to support a range of operands having bits of various widths. In at least one embodiment, floating point ALU 2022 and floating point move unit 2024 can operate on 128 bits wide packed data operands in conjunction with SIMD and multimedia instructions.

[0240] In at least one embodiment, micro-instruction schedulers 2002, 2004, 2006 schedule dependent operations prior to completion of parent load execution. In at least one embodiment, because micro-instructions can be speculatively scheduled and executed in processor 2000, processor 2000 can also include logic to handle memory misses. In at least one embodiment, if a data load in a data cache misses, there can be a dependent operation running in a pipeline that causes a scheduler to temporarily have incorrect data. In at least one embodiment, a replay mechanism tracks and re-executes instructions that use incorrect data. In at least one embodiment, dependent operations can need to be replayed and independent operations can be allowed to complete. In at least one embodiment, a scheduler and replay mechanism of at least one embodiment of a processor can also be designed to capture instruction sequences for text string compare operations.

[0241] In at least one embodiment, the term “register” can refer to an on-board processor storage location that can be used as part of an instruction that identifies an operand. In at least one embodiment, a register can be one that can be used from outside of a processor (from a programmer’s perspective). In at least one embodiment, a register can not be limited to a particular type of circuit. Rather, in at least one embodiment, a register can store data, provide data, and perform functions described herein. In at least one embodiment, registers described herein can be implemented by circuitry within a processor using a variety of different techniques, such as dedicated physical registers, physical registers allocated dynamically with register renaming, a combination of dedicated and dynamically allocated physical registers, etc. In at least one embodiment, an integer register stores 32-bit integer data. A register file of at least one embodiment also includes eight multimedia SIMD registers for packing data.

[0242] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. Details regarding inference and / or training logic 615 are provided below in conjunction with FIGS. 6L and / or 6M. FIG. 6A and / or FIG. 6B Details regarding inference and / or training logic 615 are provided below in conjunction with FIGS. 6L and / or 6M. In at least one embodiment, inference and / or training logic 615 can be incorporated in execution block 2011 and other memory or registers shown or not shown. For example, in at least one embodiment, training and / or inferencing techniques described herein can use one or more ALUs shown in execution block 2011. Further, weight parameters can be stored in on-chip or off-chip memory and / or registers (shown or not shown) that configure ALUs of execution block 2011 to perform one or more machine learning algorithms, neural network architectures, use cases, or training techniques described herein.

[0243] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. In at least one embodiment, used in conjunction with components of these figures, logic can generate one or more images of an object with an identified class removed.

[0244] FIG. 21 A deep learning application processor 2100 according to at least one embodiment is shown. In at least one embodiment, deep learning application processor 2100 uses instructions that, if executed by deep learning application processor 2100, cause deep learning application processor 2100 to perform some or all of the processes and techniques described throughout this disclosure. In at least one embodiment, deep learning application processor 2100 is an application specific integrated circuit (ASIC). In at least one embodiment, application processor 2100 performs matrix multiplication operations or is “hardwired” into hardware as a result of executing one or more instructions or both. In at least one embodiment, deep learning application processor 2100 includes, without limitation, processing clusters 2110(1)-2110(12), inter-chip links (“ICLs”) 2120(1)-2120(12), inter-chip controllers (“ICCs”) 2130(1)-2130(2), memory controllers (“Mem Ctrlrs”) 2142(1)-2142(4), high bandwidth memory physical layers (“HBM PHYs”) 2144(1)-2144(4), management controller central processing units (“management controller CPUs”) 2150, peripheral component interconnect express controllers and direct memory access blocks (“PCIe controllers and DMA”) 2170, and sixteen lane peripheral component interconnect express ports (“PCI Express x 16”) 2180.

[0245] In at least one embodiment, processing clusters 2110 can perform deep learning operations, including inferencing or prediction operations based on weight parameters calculated based on one or more training techniques, including those described herein. In at least one embodiment, each processing cluster 2110 can include, without limitation, any number and type of processor. In at least one embodiment, deep learning application processor 2100 can include any number and type of processing clusters 2100. In at least one embodiment, inter-chip links 2120 are bidirectional. In at least one embodiment, inter-chip links 2120 and inter-chip controllers 2130 enable multiple deep learning application processors 2100 to exchange information, including activation information resulting from execution of one or more neural networks embodying one or more machine learning algorithms. In at least one embodiment, deep learning application processor 2100 can include any number (including zero) and type of ICLs 2120 and ICCs 2130.

[0246] In at least one embodiment, HBM2 2140 provides a total of 32 GB of memory. HBM2 2140(i) is associated with both a memory controller 2142(i) and an HBM PHY 2144(i). In at least one embodiment, any number of HBM2s 2140 can provide any type and total amount of high bandwidth memory and can be associated with any number (including zero) and type of memory controllers 2142 and HBM PHYs 2144. In at least one embodiment, SPI, I2C, GPIO 3360, PCIe controller 2160, and DMA 2170 and / or PCIe 2180 can be replaced with any number and type of block to implement any number and type of communication standard in any technically feasible fashion.

[0247] Inference and / or training logic 1015 are used to perform inferencing and / or training operations associated with one or more embodiments. In at least one embodiment, inference and / or training logic 1015 can be used in FIG. 6A and / or FIG. 6B Details regarding inference and / or training logic 615 are provided. In at least one embodiment, deep learning application processor 2100 is used to train a machine learning model (e.g., neural network) to predict or infer information provided to deep learning application processor 2100. In at least one embodiment, deep learning application processor 2100 is used to infer or predict information based on a trained machine learning model (e.g., neural network) that has been trained by another processor or system or by deep learning application processor 2100. In at least one embodiment, processor 2100 can be used to perform one or more neural network use cases described herein.

[0248] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. In at least one embodiment, the logic can be used in place of logic illustrated in these figures to generate one or more images of objects with identified classes removed.

[0249] FIG. 22 is a block diagram of a neuromorphic processor 2200, in accordance with at least one embodiment. In at least one embodiment, neuromorphic processor 2200 can receive one or more inputs from a source external to neuromorphic processor 2200. In at least one embodiment, these inputs can be transmitted to one or more neurons 2202 within neuromorphic processor 2200. In at least one embodiment, neurons 2202 and components thereof can be implemented using circuitry or logic including one or more arithmetic logic units (ALUs). In at least one embodiment, neuromorphic processor 2200 can include, without limitation, thousands or millions of instances of neurons 2202, but any suitable number of neurons 2202 can be used. In at least one embodiment, each instance of neurons 2202 can include neuron input 2204 and neuron output 2206. In at least one embodiment, neurons 2202 can generate outputs that can be transmitted to inputs of other instances of neurons 2202. In at least one embodiment, neuron input 2204 and neuron output 2206 can be interconnected via synapses 2208.

[0250] In at least one embodiment, neurons 2202 and synapses 2208 can be interconnected such that neuromorphic processor 2200 operates to process or analyze information received by neuromorphic processor 2200. In at least one embodiment, a neuron 2202 can send out a pulse of output (or a “spike” or “peak”) when input received through neuron input 2204 exceeds a threshold value. In at least one embodiment, neuron 2202 can sum or integrate signals received at neuron input 2204. For example, in at least one embodiment, neuron 2202 can be implemented as a leaky integrate-and-fire neuron, where neuron 2202 can produce an output (or “spike”) using a transfer function such as a sigmoid or threshold function if a sum (called “membrane potential”) exceeds a threshold value. In at least one embodiment, a leaky integrate-and-fire neuron can sum signals received at neuron input 2204 into a membrane potential, and can apply a program decay factor (or leak) to reduce the membrane potential. In at least one embodiment, a leaky integrate-and-fire neuron can spike if multiple input signals are received at neuron input 2204 fast enough to exceed a threshold value (i.e., before the membrane potential decays too low to spike). In at least one embodiment, neuron 2202 can be implemented using circuitry or logic that receives input, integrates input into a membrane potential, and decays the membrane potential. In at least one embodiment, input can be averaged, or any other suitable transfer function can be used. Furthermore, in at least one embodiment, neuron 2202 can include, without limitation, comparator circuitry or logic that produces an output spike at neuron output 2206 when a result of applying a transfer function to neuron input 2204 exceeds a threshold value. In at least one embodiment, once neuron 2202 spikes, it can ignore previously received input information by, for example, resetting the membrane potential to 0 or another suitable default value. In at least one embodiment, once the membrane potential is reset to 0, neuron 2202 can resume normal operation after a suitable period of time (or refractory period).

[0251] In at least one embodiment, neurons 2202 can be interconnected by synapses 2208. In at least one embodiment, synapses 2208 can operate to transmit a signal from an output of a first neuron 2202 to an input of a second neuron 2202. In at least one embodiment, a neuron 2202 can transmit information over more than one instance of a synapse 2208. In at least one embodiment, one or more instances of neuron output 2206 can be connected through an instance of synapse 2208 to an instance of neuron input 2204 in the same neuron 2202. In at least one embodiment, an instance of a neuron 2202 that produces an output to be transmitted over an instance of a synapse 2208 can be referred to as a “presynaptic neuron” with respect to that instance of synapse 2208. In at least one embodiment, an instance of a neuron 2202 that receives input transmitted through an instance of a synapse 2208 can be referred to as a “postsynaptic neuron” with respect to that instance of synapse 2208. In at least one embodiment, with respect to various instances of synapse 2208, because an instance of a neuron 2202 can receive input from one or more instances of synapse 2208 and can also transmit output through one or more instances of synapse 2208, a single instance of a neuron 2202 can be both a “presynaptic neuron” and a “postsynaptic neuron”.

[0252] Neurons 2202 can be organized into one or more layers. Each instance of a neuron 2202 can have a neuron output 2206 that can fan out to one or more neuron inputs 2204 through one or more synapses 2208. In at least one embodiment, neuron outputs 2206 of neurons 2202 in a first layer 2210 can be connected to neuron inputs 2204 of neurons 2202 in a second layer 2212. In at least one embodiment, layer 2210 can be referred to as a “feedforward layer”. In at least one embodiment, each instance of a neuron 2202 in an instance of a first layer 2210 can fan out to each instance of a neuron 2202 in a second layer 2212. In at least one embodiment, first layer 2210 can be referred to as a “fully connected feedforward layer”. In at least one embodiment, each instance of a neuron 2202 in each instance of a second layer 2212 fans out to fewer than all instances of a neuron 2202 in a third layer 2214. In at least one embodiment, second layer 2212 can be referred to as a “sparsely connected feedforward layer”. In at least one embodiment, neurons 2202 in second layer 2212 can fan out to neurons 2202 in multiple other layers, including to neurons 2202 in (the same) second layer 2212. In at least one embodiment, second layer 2212 can be referred to as a “recurrent layer”. In at least one embodiment, neuromorphic processor 2200 can include any suitable combination of recurrent and feedforward layers, including but not limited to sparsely connected feedforward layers and fully connected feedforward layers.

[0253] In at least one embodiment, neuromorphic processor 2200 can include, without limitation, a reconfigurable interconnect architecture or a dedicated hardwired interconnect to connect synapses 2208 to neurons 2202. In at least one embodiment, neuromorphic processor 2200 can include, without limitation, circuitry or logic that, depending on a neural network topology and neuron fan-in / fan-out, allows synapses to be allocated to different neurons 2202 as needed. For example, in at least one embodiment, synapses 2208 can be connected to neurons 2202 using an interconnect structure such as a network-on-chip or through dedicated connections. In at least one embodiment, synapse interconnects and components thereof can be implemented using circuitry or logic.

[0254] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. In at least one embodiment, logic can be used with components of these figures to generate one or more images of objects having removed identified classes.

[0255] FIG. 23A processing system is shown in accordance with at least one embodiment. In at least one embodiment, system 2300 includes one or more processor(s) 2302 and one or more graphics processor(s) 2308, and can be a single processor desktop system, a multiprocessor workstation system, or a server system having many processor(s) 2302 or processor core(s) 2307. In at least one embodiment, system 2300 is a processing platform incorporated within a system on a chip (SoC) integrated circuit for use in mobile, handheld, or embedded devices.

[0256] In at least one embodiment, system 2300 can include or be incorporated within a server-based gaming platform, including a game console, a media console, a mobile gaming console, a handheld game console, or an online game console. In at least one embodiment, system 2300 is a mobile phone, a smart phone, a tablet device, or a mobile internet device. In at least one embodiment, processing system 2300 can also include or be coupled with a wearable device, such as a smart watch wearable device, smart glass device, augmented reality device, or virtual reality device. In at least one embodiment, processing system 2300 is a television or set top box device having one or more processors 2302 and a graphical interface generated by one or more graphics processors 2308.

[0257] In at least one embodiment, one or more processor(s) 2302 each include one or more processor cores 2307 to process instructions which, when executed, implement the operations for system and user software. In at least one embodiment, each of the one or more processor cores 2307 is configured to process a specific instruction set 2309. In at least one embodiment, instruction set 2309 can facilitate complex instruction set computing (CISC), reduced instruction set computing (RISC), or computing via a very long instruction word (VLIW). In at least one embodiment, processor cores 2307 can each process a different instruction set 2309, which can include instructions to facilitate emulation of other instruction sets. In at least one embodiment, processor core(s) 2307 can also include other processing devices, such as a digital signal processor (DSP).

[0258] In at least one embodiment, processor 2302 includes cache memory 2304. In at least one embodiment, processor 2302 can have a single internal cache or multiple levels of internal caches. In at least one embodiment, cache memory is shared among multiple components of processor 2302. In at least one embodiment, processor 2302 also uses an external cache (e.g., a three-level (L3) cache, or last level cache (LLC)) (not shown), which can be shared between processor cores 2307 using known cache coherency techniques. In at least one embodiment, additionally included in processor 2302 are register file 2306, which processor can include different types of registers such as integer registers, floating point registers, status registers, and instruction pointer registers to name a few. In at least one embodiment, register file 2306 can include general registers or other registers.

[0259] In at least one embodiment, one or more processors 2302 are coupled with one or more interface buses 2310 for communicating information to and from system 2300. In at least one embodiment, interface bus 2310 can be implemented using one or more of various bus technologies, such as a Peripheral Component Interconnect (PCI) bus, a Universal Serial Bus (USB), a Bluetooth® bus, a Wi-Fi® bus, a ZigBee® bus, an InfiniBand® bus, or an industry standard architecture (ISA) bus, among others. In at least one embodiment, interface bus 2310 can be a processor bus, such as a Direct Media Interface (DMI) bus, in at least one embodiment. In at least one embodiment, interface bus 2310 is not limited to DMI bus, and can include one or more Peripheral Component Interconnect buses (e.g., a PCI, a PCI Express), a memory bus, or other types of interface buses. In at least one embodiment, processor 2302 includes an integrated memory controller 2316 and platform controller hub 2330. In at least one embodiment, memory controller 2316 facilitates communication between memory devices and other components of computing system 2300, while platform controller hub 2330 provides connections to input / output (I / O) devices to local I / O bus.

[0260] In at least one embodiment, memory device 2320 can be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, flash memory device, or a phase change memory device, among others. In at least one embodiment, memory device 2320 can be a system memory of processing system 2300, to store data 2322 and instructions 2321 for use when one or more processors 2302 executes an application or process. In at least one embodiment, memory controller 2316 also couples with an optional external graphics processor 2312, which can communicate with one or more graphics processors 2308 in processors 2302 to perform graphics and media operations.

[0261] In at least one embodiment, platform controller hub 2330 enables peripherals to connect to storage device 2320 and processor 2302 via a high-speed I / O bus. In at least one embodiment, I / O peripherals include, but are not limited to, audio controller 2346, network controller 2334, firmware interface 2328, wireless transceiver 2326, touch sensors 2325, data storage device 2324 (e.g., hard disk drive, flash memory, etc.). In at least one embodiment, data storage device 2324 can connect to the storage interface via a storage interface bus, e.g., a SATA

[0262] In at least one embodiment, memory controller 2316 and instances of platform controller hub 2330 can be integrated into a discrete external graphics processor, such as external graphics processor 2312. In at least one embodiment, platform controller hub 2330 and / or memory controller 2316 can be external to one or more processor(s) 2302. For example, in at least one embodiment, system 2300 can include an external memory controller 2316 and platform controller hub 2330, which can be configured as a memory hub and peripheral hub, respectively, in a system-on-a-chip (SoC) configuration.

[0263] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. In at least one embodiment, inference and / or training logic 615 can be used in place of, or to supplement, one or more of inference and / or training logic 515 and / or 625 described herein. FIG. 6A and / or FIG. 6BDetails regarding the inference and / or training logic 615 are provided. In at least one embodiment, some or all of inference and / or training logic 615 can be incorporated with graphics processor 2300. For example, in at least one embodiment, the training and / or inference techniques described herein can use one or more ALUs embodied in graphics processor 2312. Moreover, in at least one embodiment, the inference and / or training operations described herein can be accomplished with logic other than that shown. FIG. 6A or FIG. 6B In at least one embodiment, weight parameters can be stored in on-chip or off-chip memory and / or registers (shown or not) that configure ALUs of graphics processor 2300 to perform one or more machine learning algorithms, neural network architectures, use cases, or training techniques described herein.

[0264] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. In at least one embodiment, logic can be used with components of these figures to generate one or more images of objects having removed identified classes.

[0265] FIG. 24 is a block diagram of a processor 2400 having one or more processor cores 2402A-2402N, an integrated memory controller 2414, and an integrated graphics processor 2408, according to at least one embodiment. In at least one embodiment, processor 2400 can include additional cores, up to and including an additional core 2402N represented by a dashed lined in at least one embodiment. In at least one embodiment, each processor core 2402A-2402N includes one or more internal cache units 2404A-2404N. In at least one embodiment, each processor core can also include access to one or more shared cache units 2406.

[0266] In at least one embodiment, internal cache units 2404A-2404N and shared cache unit 2406 represent a cache memory hierarchy within processor 2400. In at least one embodiment, cache memory units 2404A-2404N can include at least one level of instruction and data caches per processor core and one or more shared mid-level caches, such as Level 2 (L2), Level 3 (L3), Level 4 (L4), or other levels of cache, with the top level of cache being classified as LLC. In at least one embodiment, cache coherence logic maintains coherence between various cache units 2406 and 2404A-2404N.

[0267] In at least one embodiment, processor 2400 can also include a set of one or more bus controller units 2416 and a system agent core 2410. In at least one embodiment, one or more bus controller units 2416 manage a set of peripheral buses, such as one or more PCI or PCIe buses. In at least one embodiment, system agent core 2410 provides management functionality for various processor components. In at least one embodiment, system agent core 2410 includes one or more integrated memory controllers 2414 to manage access to various external memory devices (not shown).

[0268] In at least one embodiment, one or more processor cores 2402A-2402N include support for simultaneous multi-threading. In at least one embodiment, system agent core 2410 includes components for coordination and operation of cores 2402A-2402N during multi-threaded processing. In at least one embodiment, system agent core 2410 can additionally include a power control unit (PCU) including logic and components to regulate one or more power states of processor cores 2402A-2402N and graphics processor 2408.

[0269] In at least one embodiment, processor 2400 also includes graphics processor 2408 to perform graphics processing operations. In at least one embodiment, graphics processor 2408 couples with shared cache unit 2406 and system agent core 2410, including one or more integrated memory controllers 2414. In at least one embodiment, system agent core 2410 also includes a display controller 2411 to drive output to one or more coupled displays. In at least one embodiment, display controller 2411 can also be a separate module coupled with graphics processor 2408 via at least one interconnect, or can be integrated within graphics processor 2408.

[0270] In at least one embodiment, ring-based interconnect unit 2412 is used to couple the internal components of processor 2400. In at least one embodiment, an alternative interconnect unit can be used, such as a point-to-point interconnect, a switched interconnect, or other technology. In at least one embodiment, graphics processor 2408 couples with ring interconnect 2412 via I / O link 2413.

[0271] In at least one embodiment, I / O link 2413 represents at least one of a variety of I / O interconnects, including packaged I / O interconnects that facilitate communication between various processor components and high-performance embedded memory module 2418 (e.g., eDRAM module). In at least one embodiment, each of processor cores 2402A-2402N and graphics processor 2408 uses embedded memory module 2418 as a shared last-level cache.

[0272] In at least one embodiment, processor cores 2402A-2402N are homogeneous cores executing a common instruction set architecture. In at least one embodiment, processor cores 2402A-2402N are heterogeneous in terms of instruction set architecture (ISA), with one or more processor cores 2402A-2402N executing a common instruction set, while one or more other processor cores 2402A-2402N execute a subset of the common instruction set or a different instruction set. In at least one embodiment, processor cores 2402A-2402N are heterogeneous in terms of microarchitecture, with one or more cores having relatively high power consumption coupled to one or more power cores having lower power consumption. In at least one embodiment, processor 2400 may be implemented on one or more chips or implemented as a SoC integrated circuit.

[0273] Inference and / or training logic 615 is used to perform inference and / or training operations associated with one or more embodiments. The following is in conjunction with... FIG. 6A and / or FIG. 6B Details regarding the inference and / or training logic 615 are provided. In at least one embodiment, some or all of the inference and / or training logic 615 may be incorporated into the processor 2400. For example, in at least one embodiment, the training and / or inference techniques described herein may be used in the graphics processor 2312, one or more graphics cores 2402A-2402N, or... FIG. 24 One or more ALUs embodied in other components. Furthermore, in at least one embodiment, the inference and / or training operations described herein can use, in addition to... FIG. 6A or FIG. 6B The logic is performed in a manner other than that shown. In at least one embodiment, weight parameters may be stored in on-chip or off-chip memory and / or registers (shown or not shown), which configure the ALU of the graphics processor 2400 to execute one or more machine learning algorithms, neural network architectures, use cases, or training techniques described herein.

[0274] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. In at least one embodiment, logic can be used in conjunction with components of these figures to generate one or more images of objects having removed identified classes, in accordance with at least one embodiment described herein.

[0275] FIG. 25 is a block diagram of hardware logic of a graphics processor core 2500 in accordance with at least one embodiment described herein. In at least one embodiment, graphics processor core 2500 is included within a graphics core array. In at least one embodiment, graphics processor core 2500 (sometimes referred to as a core slice) can be one or more graphics cores within a modular graphics processor. In at least one embodiment, graphics processor core 2500 is an example of one graphics core slice, and a graphics processor as described herein can include multiple graphics core slices based on target power and performance envelopes. In at least one embodiment, each graphics core 2500 can include fixed function block 2530 coupled with multiple sub-cores 2501A-2501F (also referred to as sub-slices) that include modular blocks with general and fixed function logic.

[0276] In at least one embodiment, fixed function block 2530 includes geometry and fixed function pipeline 2536, for example, that can be shared by all of the sub-cores in graphics processor 2500 in lower performance and / or lower power graphics processor implementations. In at least one embodiment, geometry and fixed function pipeline 2536 includes a 3D fixed function pipeline, a video front-end unit, a thread generator and thread dispatcher, and a unified return buffer manager that manages a unified return buffer.

[0277] In at least one embodiment, fixed function block 2530 also includes a graphics SoC interface 2537, a graphics microcontroller 2538, and a media pipeline 2539. In at least one embodiment, graphics SoC interface 2537 provides an interface between graphics core 2500 and other processor cores within a system on a chip. In at least one embodiment, graphics microcontroller 2538 is a programmable sub-processor that is configurable to manage various functions of graphics processor 2500, including thread dispatch, scheduling, and pre-emption. In at least one embodiment, media pipeline 2539 includes logic to facilitate decoding, encoding, pre-processing, and / or post-processing of multimedia data, including image and video data. In at least one embodiment, media pipeline 2539 implements media operations via requests to compute or sample logic within sub-cores 2501-2501F.

[0278] In at least one embodiment, SoC interface 2537 enables graphics core 2500 to communicate with general application processor cores (e.g., CPUs) and / or other components within the SoC, including memory hierarchy elements such as shared last level cache, system RAM, and / or embedded on-chip or package DRAM. In at least one embodiment, SoC interface 2537 can also enable communication with fixed function devices within the SoC, such as camera imaging pipelines, and enable

[0279] In at least one embodiment, graphics microcontroller 2538 can be configured to perform various scheduling and management tasks for graphics core 2500. In at least one embodiment, graphics microcontroller 2538 can perform graphics and / or compute workload scheduling on various graphics processing engines within execution unit (EU) arrays 2502A-2502F, 2504A-2504F in sub-cores 2501A-2501F. In at least one embodiment, host software executing on CPU cores of an SoC including graphics core 2500 can submit workloads for one of graphics processing units, which invokes scheduling operations on appropriate graphics engines. In at least one embodiment, scheduling operations include determining which workload to run next, submitting a workload to a command streamer, pre-empting existing workloads running on an engine, monitoring progress of a workload, and notifying host software when a workload completes. In at least one embodiment, graphics microcontroller 2538 can also facilitate low power or idle states for graphics core 2500, providing ability to save and restore registers across low power state transitions independent of operating system and / or graphics driver software on graphics core 2500.

[0280] In at least one embodiment, graphics core 2500 can have up to N more or less modular cores than shown in FIG. 25A. For each set of N cores, graphics core 2500 can also include shared function logic 2510, shared and / or cache memory 2512, geometry / fixed function pipeline 2514, and additional fixed function logic 2516 to accelerate various graphics and compute operations. In at least one embodiment, shared function logic 2510 can include logic units (e.g., samplers, math, and / or inter-thread communication logic) that are shared by each N core within graphics core 2500. In at least one embodiment, shared and / or cache memory 2512 can be a last level cache memory for N cores 2501A-2501F within graphics core 2500 and can also be used as shared memory that can be accessed by multiple cores. In at least one embodiment, geometry / fixed function pipeline 2514 can be included instead of geometry / fixed function pipeline 2536 within fixed function block 2530 and can include the same or similar logic units.

[0281] In at least one embodiment, graphics core 2500 includes additional fixed function logic 2516 that can include various fixed function acceleration logic used by graphics core 2500. In at least one embodiment, additional fixed function logic 2516 includes an additional geometry pipeline used in position only shading. In position only shading, there are at least two geometry pipelines, while in full geometry and fixed function pipeline 2514, 2536, and cull pipeline, which is a trimmed down version of full geometry pipeline that can be included in additional fixed function logic 2516. In at least one embodiment, cull pipeline is a trimmed down version of full geometry pipeline. In at least one embodiment, full and cull pipelines can execute different instances of an application, each with separate state. In at least one embodiment, position only shading can hide long cull runs of triangles that are discarded, which can complete shading earlier in some cases. For example, in at least one embodiment, cull pipeline logic in additional fixed function logic 2516 can execute position shaders in parallel with a main application and often generate critical results faster than full pipeline because cull pipeline takes and shades position attributes of vertices without performing rasterization and rendering pixels to a frame buffer. In at least one embodiment, cull pipeline can use generated critical results to compute visibility information for all triangles regardless of whether these triangles are culled or not. In at least one embodiment, full pipeline, which can be referred to as replay pipeline in this case, can consume visibility information to skip culled triangles to only shade visible triangles that are finally passed to rasterization stage.

[0282] In at least one embodiment, additional fixed function logic 2516 can also include machine learning acceleration logic, such as fixed function matrix multiplication logic, for implementing optimizations including for machine learning training or inferencing.

[0283] In at least one embodiment, within each graphics sub-core 2501A-2501F includes a set of execution resources, which can be used to perform graphics, media, and compute operations in response to requests by graphics pipeline, media pipeline, or shader program. In at least one embodiment, graphics sub-cores 2501A-2501F include multiple arrays of execution units 2502A-2502F, 2504A-2504F, thread dispatch and inter-thread communication (TD / IC) logic 2503A-2503F, 3D (e.g., texture) samplers 2505A-2505F, media samplers 2506A-2506F, shader processors 2507A-2507F, and shared local memory (SLM) 2508A-2508F. Arrays of execution units 2502A-2502F, 2504A-2504F each include multiple execution units, which are general-purpose graphics processing units capable of performing floating-point and integer / fixed-point logic operations including graphics, media, or compute shader programs. In at least one embodiment, TD / IC logic 2503A-2503F performs local thread dispatch and thread control operations for execution units within a sub-core and facilitate

[0284] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. In at least one embodiment, inferencing and / or training operations include machine learning for implementing neural networks to execute adaptive audio coding operations described herein. FIG. 6A and / or FIG. 6BDetails regarding the inference and / or training logic 615 are provided. In at least one embodiment, portions of inference and / or training logic 615 can be incorporated into graphics processor 2510. For example, in at least one embodiment, training and / or inferencing techniques described herein can be performed using one or more ALUs embodied in graphics processor 2312, graphics microcontroller 2538, geometry and fixed function pipeline 2514 and 2536, or other logic in FIG. 25 In addition, in at least one embodiment, inferencing and / or training operations described herein can be done using logic other than that shown in FIG. 6A or FIG. 6B In at least one embodiment, weight parameters can be stored in on-chip or off-chip memory and / or registers (shown or not) that configure ALUs of graphics processor 2500 to perform one or more machine learning algorithms, neural network architectures, use cases, or training techniques described herein.

[0285] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. In at least one embodiment, logic can be used with components of these figures to generate one or more images that have removed objects of an identified class.

[0286] FIG. 26A-26B Thread execution logic 2600 illustrating an array of processing elements including graphics processor cores is shown in accordance with at least one embodiment. FIG. 26A At least one embodiment is shown in which thread execution logic 2600 is used. FIG. 26B Exemplary internal details of a processing element are shown in accordance with at least one embodiment.

[0287] As FIG. 26AAs shown in FIG. 26, in at least one embodiment, thread execution logic 2600 includes a shader processor 2602, a thread dispatcher 2604, an instruction cache 2606, a scalable execution unit array including a plurality of execution units 2608A-2608N, a sampler 2610, a data cache 2612, and a data port 2614. In at least one embodiment, the scalable execution unit array can dynamically scale by enabling or disabling one or more execution units (e.g., any of execution units 2608A, 2608B, 2608C, 2608D, through 2608N-l, and 2608N) based on a workload of a graphics processing cluster. In at least one embodiment, the scalable execution unit array is interconnected to one or more shared function units including a shared function unit 2616. In at least one embodiment, the shared function unit 2616 includes one or more shared function units (e.g., shared function units 2616A, 2616B, and 2616C) that provide support for graphics processing, including but not limited to hash operations, surface management, and cache allocation. In at least one embodiment, the shared function units 2616A-2616C are shared by the graphics processing cluster of the processing unit. In at least one embodiment, the shared function units 2616A-2616C are shared by a plurality of graphics processing clusters of a graphics processing unit. In at least one embodiment, the shared function units 2616A-2616C are shared by a plurality of graphics processing units. In at least one embodiment, the shared function units 2616A-2616C are shared by a plurality of processing units of a multiprocessor. In at least one embodiment, the shared function units 2616A-2616C are shared by a plurality of central processing units of a computing device. In at least one embodiment, the shared function units 2616A-2616C are shared by a plurality of central processing units of a computing device.

[0288] In at least one embodiment, execution units 2608A-2608N primarily execute shader programs. In at least one embodiment, shader processor 2602 processes various shader programs and dispatches execution threads associated with the shader programs to execution units 2608A-2608N via thread dispatcher 2604. In at least one embodiment, thread dispatcher 2604 includes logic to arbitrate thread initialization requests from graphics and media pipelines and instantiate requested threads on one or more of execution units 2608A-2608N. For example, in at least one embodiment, a geometry pipeline can dispatch a vertex, tessellation or geometry shader to thread execution logic for processing. In at least one embodiment, thread dispatcher 2604 can also handle runtime thread spawning requests from running shader programs.

[0289] In at least one embodiment, execution units 2608A-2608N support a wide variety of instructions that are capable of performing a wide variety of integer and floating point operations. In at least one embodiment, execution units 2608A-2608N can also include specialized support (e.g., built-in functions) for cryptographic operations and other programmable or non- programmable tasks. In at least one embodiment, execution units 2608A-2608N include dedicated hardware support for fingerprint analysis and pattern matching operations, including 128-bit SIMD fingerprinting instructions and a pattern match engine (PME) with dedicated register file and other resources.

[0290] In at least one embodiment, each of execution units 2608A-2608N operate on arrays of data elements. In at least one embodiment, a number of data elements is a “execution size,” or number of channels for the instruction. In at least one embodiment, an execution channel refers to a logical unit of execution for data element access, masking, and flow control within instructions. In at least one embodiment, a number of channels can be independent of a number of physical ALUs or FPUs for a particular graphics processor. In at least one embodiment, execution units 2608A-2608N support integer and floating-point data types.

[0291] In at least one embodiment, execution unit instruction set includes SIMD instructions. In at least one embodiment, SIMD instructions support multiple instruction execution (e.g., instruction level parallelism). In at least one embodiment, SIMD instructions 0x0F, 0x61, and 0x17 are used for packed 64 and 128 bit software visible operations. In at least one embodiment, SIMD instructions 0x0F, 0x6B, and 0x13 are used for packed 32 bit software visible operations. In at least one embodiment, various data elements can be stored as a packed data type within a register and the execution unit will process the various elements within the packed data type. In at least one embodiment, elements can be stored and accessed with in a register as follows: 64 bit registers allow 2 32-bit double-precision (64-bit) floating point data elements to be stored and processed as a single entity. 32-bit registers allow 4 single-precision (32-bit) floating point data elements to be stored and processed as a single entity. In at least one embodiment, 96 bit registers allow 3 double precision floating point elements to be stored and processed in a single entity. In at least one embodiment, 128 bit registers allow 4 single precision floating point elements to be stored and processed in a single entity. In at least one embodiment, 256 bit registers allow 8 single precision floating point elements to be stored and processed in a single entity.

[0292] In at least one embodiment, one or more execution units can be combined in a fused execution unit 2609A-2609N having thread control logic (2607A-2607N) common to the fused EU. In at least one embodiment, multiple EUs can be merged into a group of EUs. In at least one embodiment, the number of EUs in a fused group of EUs can be configured to execute a separate SIMD hardware thread. The number of EUs in a fused group of EUs can vary according to various embodiments. In at least one embodiment, each EU can execute a variety of SIMD widths including, but not limited to, SIMD8, SIMD16, and SIMD32. In at least one embodiment, each fused graphics execution unit 2609A-2609N includes at least two execution units. For example, in at least one embodiment, fused execution unit 2609A includes first EU 2608A, second EU 2608B, and thread control logic 2607A common to first EU 2608A and second EU 2608B. In at least one embodiment, thread control logic 2607A controls threads executing on fused graphics execution unit 2609A, allowing each EU within fused execution unit 2609A-2609N to use a common instruction pointer register for execution.

[0293] In at least one embodiment, one or more internal instruction caches (e.g., 2606) are included in thread execution logic 2600 to cache thread instructions for execution units. In at least one embodiment, one or more data caches (e.g., 2612) are included to cache thread data during thread execution. In at least one embodiment, a sampler 2610 is included to provide texture sampling for 3D operations and media sampling for media operations. In at least one embodiment, sampler 2610 includes specialized texture or media sampling functionality to process texture or media data during sampling prior to being provided to an execution unit.

[0294] During execution, in at least one embodiment, the graphics and media pipeline sends thread initiation requests to the thread execution logic 2600 through thread spawn and dispatch logic. In at least one embodiment, once a set of geometry objects has been processed and rasterized into pixel data, pixel processor logic (e.g., pixel shader logic, fragment shader logic, etc.) within the shader processor 2602 is invoked to further compute output information and cause the results to be written to an output surface (e.g., a color buffer, a depth buffer, a stencil buffer, etc.). In at least one embodiment, pixel or fragment shader programs provided by an application program interface (API) are executed by the pixel processor logic within the shader processor 2602. In at least one embodiment, to execute a shader program, the shader processor 2602 dispatches threads to execution units (e.g., 2608A) via a thread dispatcher 2604. In at least one embodiment, the shader processor 2602 uses texture sampling logic in a sampler 2610 to access texture data stored in a texture map in memory. In at least one embodiment, arithmetic operations on the texture data and input geometry data compute pixel color data for each geometric fragment, or discard one or more pixels for further processing.

[0295] In at least one embodiment, a data port 2614 provides a memory access mechanism for the thread execution logic 2600 to output processed data to memory for further processing on a graphics processor output pipeline. In at least one embodiment, the data port 2614 includes or is coupled with one or more cache memories (e.g., data cache 2612) to cache data for memory access via a data port.

[0296] As FIG. 26BAs shown, in at least one embodiment, graphics processing unit 2608 can include an instruction fetch unit 2637, a general register file array (GRF) 2624, an architectural register file array (ARF) 2626, a thread arbiter 2622, an issue unit 2630, a branch unit 2632, a set of SIMD floating point units (FPUs) 2634, and in at least one embodiment, a set of dedicated integer SIMD ALUs 2635. GRF 2624 and ARF 2626 include a set of general and architectural register files associated with each simultaneous hardware thread that can be active in graphics processing unit 2608. In at least one embodiment, each thread architectural state is maintained in ARF 2626, while data used during thread execution is stored in GRF 2624. In at least one embodiment, each thread’s execution state, including each thread’s instruction pointer, can be saved in thread-specific registers in ARF 2626.

[0297] In at least one embodiment, graphics processing unit 2608 has an architecture that is a combination of simultaneous multi-threading (SMT) and fine-grained interleaved multi-threading (IMT). In at least one embodiment, the architecture has a modular configuration that can be fine-tuned at design time based on a target number of simultaneous threads and a number of registers per execution unit, where execution unit resources are logically allocated for execution of multiple simultaneous threads.

[0298] In at least one embodiment, graphics processing unit 2608 can co-issue multiple instructions, each of which can be different instructions. In at least one embodiment, thread arbiter 2622 of graphics processing unit thread 2608 can dispatch an instruction to one of issue unit 2630, branch unit 2632, or SIMD FPU 2634 for execution. In at least one embodiment, each execution thread can have access to 128 general registers in GRF 2624, where each register can store 32 bytes, which can be accessed as a SIMD 8-element vector of 32-bit data elements. In at least one embodiment, each execution unit thread can have access to 4 KB in GRF 2624, although embodiments are not limited thereto, and in other embodiments can provide more or less register resources. In at least one embodiment, although the number of threads per execution unit can also vary from embodiment to embodiment, up to seven threads can be executed simultaneously. In at least one embodiment in which seven threads have access to 4 KB, GRF 2624 can store a total of 28 KB. In at least one embodiment, flexible addressing modes can allow registers to be addressed together to effectively establish wider registers or rectangular block data structures that represent strides.

[0299] In at least one embodiment, memory operations, sampler operations, and other longer-latency system communications are dispatched via a “send” instruction executed by message passing send unit 2630. In at least one embodiment, dispatch of branch instructions to a dedicated branch unit 2632 facilitates SIMD divergence and eventual convergence.

[0300] In at least one embodiment, graphics execution unit 2608 includes one or more SIMD floating point units (FPU) 2634 to perform floating point operations. In at least one embodiment, one or more FPU 2634 also support integer computation. In at least one embodiment, one or more FPU 2634 can SIMD execute up to M 32-bit floating point (or integer) operations per clock cycle, or up to 2M 16-bit integer or 16-bit floating point operations per clock cycle. In at least one embodiment, at least one FPU provides extended math capability to support high throughput of math functions previously implemented in vertex shader execution units. In at least one embodiment, there also exists a set of 8-bit integer SIMD ALUs 2635 which can be specially optimized to accelerate machine learning training and inference operations.

[0301] In at least one embodiment, an array of multiple instances of graphics execution unit 2608 can be instantiated in a graphics sub-core group (e.g., a sub-slice). In at least one embodiment, execution unit 2608 can execute instructions across multiple execution lanes. In at least one embodiment, each thread executed on graphics execution unit 2608 executes on a different lane.

[0302] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. Details regarding inference and / or training logic 615 are provided below in conjunction with FIGS. 6L and / or 6M. FIG. 6A and / or FIG. 6B Details regarding inference and / or training logic 615 are provided below in conjunction with FIGS. 6L and / or 6M. In at least one embodiment, portions or all of inference and / or training logic 615 can be incorporated in graphics processing engine 2310. Moreover, in at least one embodiment, logic other than that shown in FIGS. 6L and / or 6M can be used to perform the inferencing and / or training operations described herein. In at least one embodiment, weight parameters can be stored in on-chip or off-chip memory and / or registers (shown or not shown) that configure ALUs of execution logic 2600 to perform one or more machine learning algorithms, neural network architectures, use cases, or training techniques introduced herein. FIG. 6A or FIG. 6B Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. In at least one embodiment, logic can be used with components of these figures to generate one or more images that have had objects of an identified class removed.

[0303] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. In at least one embodiment, logic can be used with components of these figures to generate one or more images that have had objects of an identified class removed.

[0304] FIG. 27 A parallel processing unit (“PPU”) 2700, in accordance with at least one embodiment, is shown. In at least one embodiment, PPU 2700 is configured with machine-readable code that, if executed by PPU 2700, causes PPU 2700 to perform some or all of the processes and techniques described throughout this disclosure. In at least one embodiment, PPU 2700 is a multi-threaded processor implemented on one or more integrated circuit devices and utilizes multi-threading as a latency-hiding technique designed to process computer-readable instructions (also referred to as machine-readable instructions or simply instructions) that are executed in parallel on multiple threads. In at least one embodiment, a thread refers to an execution thread and is an instance of a set of instructions configured to be executed by PPU 2700. In at least one embodiment, PPU 2700 is a graphics processing unit (“GPU”) configured to implement a graphics rendering pipeline for processing three-dimensional (“3D”) graphics data in order to generate two-dimensional (“2D”) image data for display on a display device such as a liquid crystal display (“LCD”) device. In at least one embodiment, PPU 2700 is used to perform computations such as linear algebraic operations and machine learning operations. FIG. 27 An example parallel processor is shown for illustrative purposes only and is to be construed as a non-limiting example of a processor architecture contemplated within the scope of this disclosure and can be supplemented and / or replaced by any appropriate processor.

[0305] In at least one embodiment, one or more PPUs 2700 are configured to accelerate high performance computing (“HPC”), data center, and machine learning applications. In at least one embodiment, PPU 2700 is configured to accelerate deep learning systems and applications including the following non-limiting examples: autonomous vehicle platforms, deep learning, high-precision voice, image, and text recognition systems, intelligent video analytics, molecular simulations, drug discovery, disease diagnosis, weather forecasting, big data analytics, astronomy, molecular dynamics simulations, financial modeling, robotics, factory automation, real-time language translation, online search optimization, and personalized user recommendations, among others.

[0306] In at least one embodiment, PPU 2700 includes, without limitation, input / output (“I / O”) units 2706, front-end units 2710, scheduler units 2712, work distribution units 2714, hub 2716, crossbar (“Xbar”) 2720, one or more general processing clusters (“GPCs”) 2718, and one or more partition units (“memory partition units”) 2722. In at least one embodiment, PPU 2700 connects with one or more host processors, or other PPUs 2700, via one or more high-speed GPU interconnects (“GPU interconnects”) 2708. In at least one embodiment, PPU 2700 connects with host processors, and other peripherals, via interconnect 2702. In at least one embodiment, PPU 2700 connects to a local memory comprising one or more memory devices (“memory”) 2704. In at least one embodiment, memory devices 2704 include, without limitation, one or more dynamic random access memory (“DRAM”) devices. In at least one embodiment, one or more DRAM devices are configured and / or configurable as high bandwidth memory (“HBM”) subsystems, with multiple DRAM dies stacked

[0307] In at least one embodiment, high-speed GPU interconnect 2708 can refer to a link-based parallel computer bus that systems use to scale, and includes one or more PPUs 2700 in conjunction with one or more central processing units (“CPUs”), supports cache coherence between PPUs 2700 and CPUs, and CPU mastering. In at least one embodiment, high-speed GPU interconnect 2708 transports data and / or commands between hub 2716 and other units of PPU 2700, such as one or more copy engines, video encoders, video decoders, power management units, and / or other components not explicitly shown in FIG. 27. FIG. 27

[0308] In at least one embodiment, I / O units 2706 are configured to facilitate communications between PPU 2700 and one or more other devices connected to system bus 2702. In at least one embodiment, I / O units 2706 include, without limitation, one or more of a peripheral component interconnect (“PCI”) bridge, a wireless transceiver, a Bluetooth transceiver, and / or other communication interface. FIG. 27 ​The I / O units 2706 send and receive communications (e.g., commands, data) to and from the system bus 2702. In at least one embodiment, the I / O units 2706 communicate directly with the host processor(s) via the system bus 2702 or through one or more intermediate devices such as a memory bridge. In at least one embodiment, the I / O units 2706 can communicate with one or more other processors (e.g., one or more PPUs 2700) via the system bus 2702. In at least one embodiment, the I / O units 2706 implement a Peripheral Component Interconnect Express (“PCIe”) interface for communications over a PCIe bus. In at least one embodiment, the I / O units 2706 implement an interface for communications with external devices.

[0309] In at least one embodiment, the I / O units 2706 decode packets received via the system bus 2702. In at least one embodiment, at least some packets represent commands configured to cause the PPU 2700 to perform various operations. In at least one embodiment, the I / O units 2706 send decoded commands to various other units of the PPU 2700 as designated by the commands. In at least one embodiment, commands are sent to the front-end unit 2710 and / or to the hub 2716 or other units of the PPU 2700 such as one or more copy engines, video encoders, video decoders, power management units, etc. FIG. 27 In at least one embodiment, the I / O units 2706 are not explicitly shown in FIG. 27. In at least one embodiment, the I / O units 2706 are configured to route communications between various logical units of the PPU 2700.

[0310] In at least one embodiment, a program executed by the host processor encodes a command stream in a buffer that provides a workload to the PPU 2700 for processing. In at least one embodiment, the workload includes instructions and data to be processed by those instructions. In at least one embodiment, the buffer is a region in memory that is accessible (e.g., read / write) by both the host processor and the PPU 2700 - the host interface unit can be configured to access memory requests transmitted via the I / O units 2706 over the system bus 2702 to connect to the buffer in system memory. In at least one embodiment, the host processor writes the command stream to the buffer and then sends a pointer to a start of the command stream to the PPU 2700 so that the front-end unit 2710 receives the one or more command stream pointers and manages the one or more command streams, reading commands from the command stream and forwarding the commands to various units of the PPU 2700.

[0311] In at least one embodiment, front-end unit 2710 is coupled to a scheduler unit 2712 which configures various GPCs 2718 to process tasks defined by one or more command streams. In at least one embodiment, scheduler unit 2712 is configured to track state information related to various tasks managed by scheduler unit 2712, where state information can indicate which task is assigned to which GPC 2718, whether a task is active or inactive, a priority associated with a task, etc. In at least one embodiment, scheduler unit 2712 manages multiple tasks that are executed on one or more GPCs 2718.

[0312] In at least one embodiment, scheduler unit 2712 is coupled to a work distribution unit 2714, which is configured to dispatch tasks for execution on GPCs 2718. In at least one embodiment, work distribution unit 2714 tracks a number of scheduled tasks received from scheduler unit 2712 and work distribution unit 2714 manages a pending task pool and an active task pool for each GPC 2718. In at least one embodiment, the pending task pool includes a number of slots (e.g., 32 slots) that hold tasks assigned to be processed by a particular GPC 2718; the active task pool can include a number of slots (e.g., 4 slots) for tasks that are actively being processed by a GPC 2718, such that as one task completes execution on a GPC 2718, that task is evicted from the active task pool for that GPC 2718 and one of the other tasks from the pending task pool is selected and scheduled for execution on the GPC 2718. In at least one embodiment, if an active task is idle, for example, while waiting for a data dependency to resolve, the active task is evicted from the GPC 2718 and returned to the pending task pool while another task is selected from the pending task pool and scheduled for execution on the GPC 2718.

[0313] In at least one embodiment, work distribution unit 2714 communicates with one or more GPCs 2718 via XBar 2720. In at least one embodiment, XBar 2720 is an interconnect network that couples many units of PPU 2700 to other units of PPU 2700 and can be configured to couple work distribution unit 2714 to a particular GPC 2718. In at least one embodiment, other units of one or more PPUs 2700 can also be connected to XBar 2720 via hub 2716.

[0314] In at least one embodiment, tasks are managed by a scheduler unit 2712 and dispatched to one of GPCs 2718 by a work distribution unit 2714. GPCs 2718 are configured to process tasks and generate results. In at least one embodiment, results can be consumed by other tasks within GPC 2718, routed to different GPCs 2718 via XBar 2720 or stored in memory 2704. In at least one embodiment, results can be written to memory 2704 via a partition unit 2722, which implements a memory interface for reading and writing data to memory 2704. In at least one embodiment, results can be transmitted to another PPU 2704 or CPU via a high-speed GPU interconnect 2708. In at least one embodiment, PPU 2700 includes, without limitation, U partition units 2722 equal to a number of separate and distinct memory devices 2704 coupled to PPU 2700. In at least one embodiment, partition unit 2722 is described in greater detail below in conjunction with FIG. 29 In more detail.

[0315] In at least one embodiment, a host processor executes a driver core that implements an application programming interface (API) that enables one or more applications executing on a host processor to schedule operations to be performed on PPU 2700. In one embodiment, multiple compute applications are executed simultaneously by PPU 2700 and PPU 2700 provides isolation, quality-of-service (“QoS”), and independent address spaces for multiple compute applications. In at least one embodiment, an application generates instructions (e.g., in the form of API calls) that cause a driver core to generate one or more tasks for execution by PPU 2700 and driver core outputs tasks to one or more streams processed by PPU 2700. In at least one embodiment, each task includes one or more related thread groups, which can be referred to as warps. In at least one embodiment, a warp includes multiple related threads (e.g., 32 threads) that can be executed in parallel. In at least one embodiment, cooperating threads can refer to a plurality of threads that include instructions for performing a task and exchanging data via shared memory. In at least one embodiment, cooperating threads are described in greater detail below in conjunction with FIG. 29 Threads and cooperating threads are described in more detail in accordance with at least one embodiment.

[0316] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. Inferences and / or training can be facilitated by one or more components described in conjunction with FIG. 6, depending on the embodiment. FIG. 6A and / or FIG. 6BDetails regarding the inference and / or training logic 615 are provided. In at least one embodiment, a deep learning application processor is used to train machine learning models, such as neural networks, to predict or infer information provided to PPU 2700. In at least one embodiment, PPU 2700 is used to infer or predict information based on a trained machine learning model (e.g., neural network) that has been trained by another processor or system or by PPU 2700. In at least one embodiment, PPU 2700 can be used to perform one or more neural network use cases described herein.

[0317] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. In at least one embodiment, logic can be used in conjunction with components of these figures to generate one or more images of objects having removed identified classes.

[0318] FIG. 28 A general processing cluster (“GPC”) 2800, according to at least one embodiment, is shown. In at least one embodiment, GPC 2800 is a GPC 2718 of FIG. 27. In at least one embodiment, each GPC 2800 includes, without limitation, a plurality of hardware units for processing tasks and each GPC 2800 includes, without limitation, a pipeline manager 2802, a pre-raster operations unit (“pre-ROP”) 2804, a raster engine 2808, a work distribution crossbar (“WDX”) 2816, a memory management unit (“MMU”) 2818, one or more Data Processing Clusters (“DPCs”) 2806, and any suitable combination of such components. FIG. 27

[0319] ​In at least one embodiment, operation of GPC 2800 is controlled by a pipeline manager 2802. In at least one embodiment, pipeline manager 2802 manages configuration of one or more DPCs 2806 for processing tasks allocated to GPC 2800. In at least one embodiment, pipeline manager 2802 configures at least one of one or more DPCs 2806 to implement at least a portion of a graphics rendering pipeline. In at least one embodiment, DPC 2806 is configured to execute vertex shader programs while pipeline manager 2802 is configured to route data from a work distribution unit to an appropriate logical unit within GPC 2800, and in at least one embodiment, some data packets can be routed to a fixed function hardware unit in pre-ROP 2804 and / or raster engine 2808 while other data packets are routed to a DPC 2806 for processing by a shader program executed by either a fragment shader unit or the SM 2814. In at least one embodiment, pipeline manager 2802 configures at least one of DPCs 2806 to implement a neural network model and / or compute pipeline.

[0320] In at least one embodiment, pre-ROP unit 2804 is configured to route data generated by raster engine 2808 and DPCs 2806 to a raster operations (“ROP”) unit in a partition unit 2722 in at least one embodiment, as described above in connection with FIG. 27. FIG. 27 In more detail, in at least one embodiment, pre-ROP unit 2804 is configured to perform optimizations for color blending, organize pixel data, perform address translations, and so forth. In at least one embodiment, raster engine 2808 includes, without limitation, a number of fixed function hardware units configured to perform various raster operations, and in at least one embodiment, raster engine 2808 includes, without limitation, a setup engine, a coarse raster engine, a cull engine, a clip engine, a fine raster engine, a tile aggregation engine, and any suitable combinations thereof. In at least one embodiment, the setup engine receives transformed vertices and generates a plane equation associated with geometric primitives defined by the vertices; the plane equation is communicated to the coarse raster engine to generate coverage information (e.g., x, y coverage masks for tiles) of the primitive; output of the coarse raster engine is communicated to the cull engine where fragments associated with primitives that fail a z-test are culled, and to the clip engine where fragments that are outside the view frustum are clipped. In at least one embodiment, the culled and clipped fragments are passed to the fine raster engine to generate attributes for pixel fragments based on the plane equation generated by the setup engine. In at least one embodiment, output of raster engine 2808 includes fragments that are to be processed by any appropriate entity, such as by a fragment shader program implemented within a DPC 2806.

[0321] In at least one embodiment, each DPC 2806 included in the GPC 2800 includes, but is not limited to, an M-pipeline controller (“MPC”) 2810; a primitive engine 2812; one or more SMs 2814; and any suitable combination thereof. In at least one embodiment, the MPC 2810 controls the operation of the DPC 2806, routing packets received from the pipeline manager 2802 to the appropriate units within the DPC 2806. In at least one embodiment, packets associated with vertices are routed to the primitive engine 2812, which is configured to retrieve vertex attributes associated with vertices from memory; conversely, packets associated with shader programs may be sent to the SM 2814.

[0322] In at least one embodiment, the SM 2814 includes, but is not limited to, a programmable streaming processor configured to process tasks represented by multiple threads. In at least one embodiment, the SM 2814 is multithreaded and configured to execute multiple threads (e.g., 32 threads) from a specific thread group concurrently, and implements a Single Instruction, Multiple Data (“SIMD”) architecture, wherein each thread in a group of threads (e.g., a thread bundle) is configured to process different datasets based on the same instruction set. In at least one embodiment, all threads in the thread group execute the same instructions. In at least one embodiment, the SM 2814 implements a Single Instruction, Multiple Thread (“SIMT”) architecture, wherein each thread in a group of threads is configured to process different datasets based on the same instructions, but wherein individual threads in the thread group are allowed to diverge during execution. In at least one embodiment, a program counter, call stack, and execution state are maintained for each thread bundle, thereby achieving concurrency between the thread bundle and serial execution within the thread bundle when threads in the thread bundle diverge. In another embodiment, a program counter, call stack, and execution state are maintained for each individual thread, thereby ensuring equal concurrency among all threads within and between thread bundles. In at least one embodiment, an execution state is maintained for each individual thread, and threads executing the same instructions can be converged and executed in parallel to improve efficiency. At least one embodiment of the SM 2814 is described in more detail below.

[0323] In at least one embodiment, the MMU 2818 is integrated with the GPC 2800 and memory partitioning unit (e.g., FIG. 27 The MMU 2818 provides an interface between partition units 2722 and provides virtual address to physical address translation, memory protection, and memory request arbitration. In at least one embodiment, the MMU 2818 provides one or more translation back buffers (“TLBs”) for performing virtual address to physical address translation in memory.

[0324] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. Details regarding inference and / or training logic 615 are provided below in conjunction with FIGS. 6L and / or 6M. FIG. 6A and / or FIG. 6B Details regarding inference and / or training logic 615 are provided below in conjunction with FIGS. 6L and / or 6M. In at least one embodiment, deep learning application processor is used to train machine learning models, such as neural networks, to predict or infer information provided to GPC 2800. In at least one embodiment, GPC 2800 is used to infer or predict information based on a machine learning model (e.g., neural network) that has been trained by another processor or system or GPC 2800. In at least one embodiment, GPC 2800 can be used to perform one or more neural network use cases described herein.

[0325] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. In at least one embodiment, logic can be used in conjunction with components of these figures to generate one or more images of objects having removed identified classes.

[0326] FIG. 29 A memory partition unit 2900 of a parallel processing unit (“PPU”) according to at least one embodiment is shown. In at least one embodiment, memory partition unit 2900 includes, without limitation, a raster operations (“ROP”) unit 2902; a level two (“L2”) cache 2904; a memory interface 2906; and any suitable combination thereof. In at least one embodiment, memory interface 2906 is coupled to a memory. In at least one embodiment, memory interface 2906 can implement a 32-, 64-, 128-, 1024-bit data bus implementation, or the like, for high-speed data transfer. In at least one embodiment, a PPUs include U memory interfaces 2906, one for each pair of partition units 2900, with each pair of partition units 2900 connected to a corresponding memory device. For example, in at least one embodiment, a PPUs can be connected to up to Y memory devices, such as a high bandwidth memory stack or graphics double data rate version 5 synchronous dynamic random access memory (“GDDR5 SDRAM”).

[0327] In at least one embodiment, memory interface 2906 implements a High-Bandwidth Memory Second Generation (“HBM2”) memory interface, and Y is equal to half of U. In at least one embodiment, HBM2 memory stacks are located on the same physical package as PPU, which can provide substantial power savings and area compared to a conventional GDDR5 SDRAM system. In at least one embodiment, each HBM2 stack includes, without limitation, four memory dies, and Y is equal to 4, each HBM2 stack includes two 128-bit channels per die for a total of 8 channels and a data bus width of 1024 bits. In at least one embodiment, memory supports a Single Error Correction Double Error Detection (“SECDED”) error-correcting code (“ECC”) to protect data. In at least one embodiment, ECC provides higher reliability for compute applications that are sensitive to data corruption.

[0328] In at least one embodiment, PPU implements a multi-level memory hierarchy. In at least one embodiment, memory partition unit 2900 supports a unified memory to provide a single unified virtual address space for central processing unit (“CPU”) and PPU memory, enabling data sharing between virtual memory systems. In at least one embodiment, frequency of PPU accesses to memory located on other processors is tracked to ensure that memory pages are moved to physical memory of PPU that accesses pages more frequently. In at least one embodiment, high-speed GPU interconnect 2708 supports address translation services, which allow PPU to directly access CPU pages and provide full access to CPU memory by PPU.

[0329] In at least one embodiment, a copy engine transfers data between multiple PPU or between a PPU and a CPU. In at least one embodiment, copy engine can generate a page fault for an address that is not mapped into a page table, and memory partition unit 2900 then services the page fault by mapping an address into a page table before copy engine performs a transfer. In at least one embodiment, a fixed (i.e., non-pageable) amount of memory is operated for multiple copy engine operations between multiple processors, thereby substantially reducing available memory. In at least one embodiment, in case of a hardware page fault, an address can be passed to a copy engine without considering whether a memory page is resident, and a copy process is transparent.

[0330] According to at least one embodiment, data from FIG. 27Data from memory 2704 or other system memory is fetched by memory partition units 2900 and stored in L2 cache 2904, which is on-chip and shared by various GPCs. In at least one embodiment, each memory partition unit 2900 includes, without limitation, at least a portion of an L2 cache associated with a corresponding memory device. In at least one embodiment, lower level caches are implemented in various units within the GPCs. In at least one embodiment, each SM 2814 can implement a level one (“Ll”) cache private to a particular SM 2814, where the Ll cache is a private memory accessible only to that SM 2814 for high bandwidth storage of temporary variables and instructions. In at least one embodiment, L2 cache 2904 is coupled to the memory interface 2906 and XBar 2720.

[0331] In at least one embodiment, ROP unit 2902 performs graphics raster operations including, without limitation, color compression, pixel blending, and the like. In at least one embodiment, ROP unit 2902 implements depth tests in conjunction with the raster engine 2808, receiving depth values for sample positions associated with pixel fragments from a culling engine of the raster engine 2808. In at least one embodiment, a respective depth test depth in a depth buffer at a sample position associated with a fragment is performed. In at least one embodiment, if the fragment passes the depth test for the sample position, ROP unit 2902 updates the depth buffer and transmits a result of the depth test to the raster engine 2808. It will be appreciated that a number of partition units 2900 can be different than a number of GPCs, and thus each ROP unit 2902 can be coupled to each GPC in at least one embodiment. In at least one embodiment, ROP unit 2902 tracks groups received from different GPCs and determines to which a result generated by ROP unit 2902 is routed through XBar 2720.

[0332] FIG. 30 A streaming multiprocessor (“SM”) 3000, in accordance with at least one embodiment, is shown. In at least one embodiment, SM 3000 is a FIG. 28SM 2814. In at least one embodiment, SM 3000 includes, without limitation, an instruction cache 3002; one or more scheduler units 3004; a register file 3008; one or more processing cores (“cores”) 3010; one or more special-function units (“SFUs”) 3012; one or more load / store units (“LSUs”) 3014; an interconnect network 3016; shared memory / level-one (“LI”) cache 3018; and any suitable combination thereof. In at least one embodiment, a work distribution unit dispatches tasks for execution on general processing clusters (“GPCs”) of parallel processing units (“PPUs”) and each task is assigned a specific data processing cluster (“DPC”) within a GPC and, if task is associated with a shader program, to one of SMs 3000. In at least one embodiment, scheduler units 3004 receive tasks from work distribution unit and manage scheduling of instruction dispatch to one or more thread blocks of SM 3000. In at least one embodiment, scheduler units 3004 schedule thread blocks to be executed to be executed as warps of parallel threads, with each thread block being assigned at least one warp. In at least one embodiment, each warp executes a thread. In at least one embodiment, scheduler units 3004 manage a plurality of different thread blocks, assign warps to different thread blocks, and then dispatch instructions from different ones of the plurality of different thread blocks to various functional units (e.g., processing cores 3010, SFUs 3012, and LSUs 3014) in each clock cycle.

[0333] In at least one embodiment, a cooperative group can refer to a programming model for organizing groups of communication threads that allows developers to express the granularity at which threads are communicating, enabling richer, more efficient parallel decomposition. In at least one embodiment, a cooperative launch API supports synchronization between thread blocks to execute parallel algorithms. In at least one embodiment, an application of a conventional programming model provides a single, simple construct for synchronizing cooperative threads: a barrier across all threads of a thread block (e.g., a syncthreads() function). However, in at least one embodiment, a programmer can define thread groups at less than a thread block granularity and synchronize within defined groups to achieve higher performance, design flexibility, and software reuse in the form of collective group-wide function interfaces. In at least one embodiment, a cooperative group enables a programmer to explicitly define thread groups at sub-block (i.e., down to a single thread) and multi-block granularity and perform collective operations, such as synchronizing threads in a cooperative group. In at least one embodiment, this programming model supports clean composition across software boundaries, such that library and utility functions can safely synchronize in their local environment without having to make assumptions about convergence. In at least one embodiment, a cooperative group primitive enables new patterns of cooperative parallelism, including but not limited to producer-consumer parallelism, opportunistic parallelism, and global synchronization across a grid of thread blocks.

[0334] In at least one embodiment, a scheduler unit 3006 is configured to send instructions to one or more of the functional units, and a scheduler unit 3004 includes, without limitation, two scheduler units 3006 that enable two different instructions from the same warp to be scheduled in each clock cycle. In at least one embodiment, each scheduler unit 3004 includes a single scheduler unit 3006 or an additional scheduler unit 3006.

[0335] In at least one embodiment, each SM 3000 includes, without limitation, a register file 3008 that provides a set of registers for functional units of the SM 3000. In at least one embodiment, register file 3008 is partitioned between functional units, with each functional unit being allocated a dedicated portion of the register file 3008. In at least one embodiment, register file 3008 is partitioned between different warps executed by the SM 3000 and register file 3008 provides temporary storage for operands of the data paths connected to the functional units. In at least one embodiment, each SM 3000 includes, without limitation, a plurality L of processing cores 3010. In at least one embodiment, SM 3000 includes, without limitation, a large number (e.g., 128 or more) of diverse processing cores 3010. In at least one embodiment, each processing core 3010 includes, without limitation, a fully-pipelined, single-precision, double-precision, and / or mixed precision processing unit that includes, without limitation, a floating point arithmetic logic unit and integer arithmetic logic unit. In at least one embodiment, floating point arithmetic logic units implement IEEE 754-2008 standard for floating point arithmetic. In at least one embodiment, processing cores 3010 include, without limitation, 64 single-precision (32-bit) floating point cores, 64 integer cores, 32 double-precision (64-bit) floating point cores, and 8 tensor cores.

[0336] In at least one embodiment, one or more tensor cores are included in processing cores 3010. In at least one embodiment, tensor cores are configured to perform deep learning matrix arithmetic, such as convolution operations for neural network training and inferencing. In at least one embodiment, each tensor core operates on 4x4 matrices and performs matrix multiplication and accumulation operations D = A x B + C, where A, B, C, and D are 4x4 matrices.

[0337] In at least one embodiment, matrix multiplication inputs A and B are 16-bit floating point matrices, and accumulation matrices C and D are 16-bit floating point or 32-bit floating point matrices. In at least one embodiment, a tensor core performs 32-bit floating point accumulation operations on 16-bit floating point input data. In at least one embodiment, 16-bit floating point multiplication uses 64 operations and results in a full precision product, which is then accumulated with other intermediate products using 32-bit floating point addition for 4x4x4 matrix multiplication. In at least one embodiment, tensor cores are used to perform larger two-dimensional or higher dimensional matrix operations composed of these smaller elements. In at least one embodiment, an API such as CUDA 9 C++ API exposes specialized matrix load, matrix multiply and accumulate, and matrix store operations to efficiently use tensor cores from a CUDA-C++ program. In at least one embodiment, at CUDA level, a warp level interface assumes a 16x16 size matrix across all 32 warp threads.

[0338] In at least one embodiment, each SM 3000 includes, without limitation, M SFUs 3012 to perform special functions (e.g., certain math functions, exponentials, logarithms, etc.). In at least one embodiment, SFUs 3012 include, without limitation, tree traversal units configured to traverse a hierarchical tree data structure. In at least one embodiment, SFUs 3012 include, without limitation, texture units configured to perform texture mapping filtering operations. In at least one embodiment, texture units are configured to load a texture map (e.g., a 2D array of texture pixels) from memory and sample the texture map to produce sampled texture values for use by a shader program executed by SM 3000. In at least one embodiment, texture maps are stored in shared memory / L1 cache 3018. In at least one embodiment, texture units use mip-maps (e.g., different levels of detail for a texture map) to perform texture operations such as filtering operations according to at least one embodiment. In at least one embodiment, each SM 3000 includes, without limitation, two texture units.

[0339] In at least one embodiment, each SM 3000 includes, without limitation, N LSUs 3014 that implement load and store operations between shared memory / L1 cache 3018 and register file 3008. In at least one embodiment, each SM 3000 includes, without limitation, interconnect network 3016 that connects each of the functional units to register file 3008 and connects LSUs 3014 to register file 3008 and shared memory / L1 cache 3018. In at least one embodiment, interconnect network 3016 is a cross-bar switch that can be configured to connect any function unit to any register in register file 3008, and to connect LSUs 3014 to registers in register file 3008 and to memory locations in shared memory / L1 cache 3018.

[0340] In at least one embodiment, shared memory / L1 cache 3018 is an array of on-chip memory that, in at least one embodiment, allows data storage and communication between SMs 3000 and between threads within SM 3000 and graphics engines. In at least one embodiment, shared memory / L1 cache 3018 includes, without limitation, 128 KB of storage and is located in a path from SM 3000 to partition units. In at least one embodiment, shared memory / L1 cache 3018 is used for caching reads and writes, in at least one embodiment. In at least one embodiment, one or more of shared memory / L1 cache 3018, L2 cache, and memory are backing stores.

[0341] In at least one embodiment, combining data cache and shared memory functionality into a single memory block provides improved performance for both types of memory accesses. In at least one embodiment, capacity is used by programs that do not use shared memory or used as a cache, e.g., if shared memory is configured to use half of capacity, and textures and load / store operations can use remaining capacity. According to at least one embodiment, integration within shared memory / L1 cache 3018 enables shared memory / L1 cache 3018 to be used as a high-throughput pipeline for streaming data, while providing high-bandwidth and low-latency access to frequently reused data. In at least one embodiment, when configured for general-purpose parallel computation, a simpler configuration can be used compared to graphics processing. In at least one embodiment, fixed function graphics processing units are bypassed, creating a more straightforward programming model. In at least one embodiment, in a general-purpose parallel computation configuration, work distribution unit assigns and distributes blocks of threads directly to DPCs. In at least one embodiment, threads in a block execute the same program, use a unique thread ID in a computation to ensure that each thread generates a unique result, use SM 3000 to execute the program and perform the computation, use shared memory / L1 cache 3018 to communicate between threads, and use LSU 3014 to read and write global memory through shared memory / L1 cache 3018 and memory partition unit. In at least one embodiment, when configured for general-purpose parallel computation, SM 3000 writes to scheduler unit 3004 commands that can be used to launch new work on DPCs.

[0342] In at least one embodiment, PPU is included in a desktop computer, laptop computer, tablet computer, server computer, supercomputer, smart- phone (e.g., a wireless, hand-held device), personal digital assistant (“PDA”), digital camera, vehicle, head mounted display, hand-held electronic device, etc. or is coupled to such devices. In at least one embodiment, PPU is implemented on a single semiconductor

[0343] In at least one embodiment, PPU can be included on a graphics card that includes one or more memory devices. This graphics card can be configured to interface with a PCIe slot on a motherboard of a desktop computer. In at least one embodiment, this PPU can be an integrated graphics processing unit (“iGPU”) that is included in a chipset of a motherboard.

[0344] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. Details regarding inference and / or training logic 615 are provided below in conjunction with FIGS. 6 A and / or 6B. FIG. 6A and / or FIG. 6B Details regarding inference and / or training logic 615 are provided below in conjunction with FIGS. 6 A and / or 6B. In at least one embodiment, deep learning application processor is used to train machine learning models, such as neural networks, to predict or infer information provided to SM 3000. In at least one embodiment, SM 3000 is used to infer or predict information based on a machine learning model (e.g., neural network) that has been trained by another processor or system or by SM 3000. In at least one embodiment, SM 3000 can be used to perform one or more neural network use cases described herein.

[0345] Inference and / or training logic 615 are used to perform inferencing and / or training operations associated with one or more embodiments. In at least one embodiment, the logic can be used in conjunction with components of these figures to generate one or more images of objects with identified classes removed.

[0346] In at least one embodiment, a single semiconductor platform can refer to a sole unitary integrated circuit or chipset that can contain one or more processors, memory, input / output circuitry, and various other components of a computing device. In at least one embodiment, a plurality of semiconductor platforms can be used in a system-on-a-chip. In at least one embodiment, semiconductor platforms can also be used in mobile devices, personal computers, desktop computers, servers, and other types of devices. In at least one embodiment, a single semiconductor platform can be used in a multi-core processing system.

[0347] In at least one embodiment, computer programs in the form of executable code or computer control logic algorithms in machine-readable executable code or computer control logic algorithms are stored in main memory 1004 and / or secondary storage. In at least one embodiment, if executed by one or more processors, the computer programs enable system 1000 to perform various functions. In at least one embodiment, memory 1004, storage, and / or any other storage is a possible example of computer-readable media. In at least one embodiment, secondary storage can refer to any suitable storage device or system for storing data, such as a hard disk drive and / or a removable storage drive, representing a floppy disk drive, a magnetic tape drive, a compact disk drive, a digital versatile disk (DVD) drive, a recording device, a universal serial bus (USB) flash memory, and / or the like. In at least one embodiment, the architecture and / or functionality of the various previous figures are implemented in the environment of CPU 1002; parallel processing system 1012; integrated circuit that can have at least portions of the capabilities of both CPU 1002; parallel processing system 1012; a chipset (e.g., a group of integrated circuits designed to work together as a unit and sold as a unit); and any appropriate combinations thereof.

[0348] In at least one embodiment, the architecture and / or functionality of the various previous figures is implemented in the environment of a general purpose computer system, a game console system, a special purpose system for entertainment purposes, a system for a mobile device, and / or other systems. In at least one embodiment, computer system 1000 can take the form of a desktop computer, laptop computer, tablet computer, server computer, super computer, smart telephone (e.g., wireless, hand held device), personal digital assistant (“PDA”), digital cameras, vehicle, head mounted display, hand held electronic device, mobile telephone device, television, workstation, game console, embedded system, and / or any other type of logic.

[0349] In at least one embodiment, parallel processing system 1012 includes, without limitation, a plurality of parallel processing units (“PPUs”) 1014 and associated memory 1016. In at least one embodiment, PPUs 1014 are connected to a host processor or other peripheral devices via an interconnect 1018 and switch 1020 or multiplexer. In at least one embodiment, parallel processing system 1012 allocates computational tasks to PPUs 1014 that can be parallelized, for example, as part of a distribution of computational tasks across multiple graphics processing unit (“GPU”) thread blocks. In at least one embodiment, memory is shared and accessed by some or all of PPUs 1014 (e.g., for read and / or write access), although such shared memory can incur a performance penalty with respect to use of local memory and registers resident on PPUs 1014. In at least one embodiment, operations of PPUs 1014 are synchronized by use of commands such as __syncthreads(), where all threads in a block (e.g., executing across multiple PPUs 1014) reach a certain code execution point before proceeding.

[0350] Virtualized computing platform

[0351] Embodiments are disclosed relating to a virtualized computing platform for advanced computing, such as image inference and image processing. Referring to FIG. 31which is an example data flow diagram of a process 3100 to generate and deploy image processing and inference pipelines, in accordance with at least one embodiment. In at least one embodiment, process 3100 can be deployed for use with imaging devices, processing devices, genomics devices, genetic sequencing devices, radiology devices, and / or other device types at a facility 3102, such as a medical facility, hospital, medical institution, clinic, research or diagnostic lab, etc. In at least one embodiment, process 3100 can be deployed to perform genomics analysis and inference on sequencing data. Examples of genomic analysis that can be performed using systems and methods described herein include, but are not limited to, variant calling, mutation detection, and gene expression quantification. Process 3100 can be performed within a training system 3104 and / or a deployment system 3106. In at least one embodiment, training system 3104 can be used to perform training, deployment, and implementation of machine learning models (e.g., neural networks, object detection algorithms, computer vision algorithms, etc.) for use in deployment system 3106. In at least one embodiment, deployment system 3106 can be configured to offload processing and computing resources between distributed computing environments in order to reduce infrastructure requirements at facility 3102. In at least one embodiment, deployment system 3106 can provide a streamlined platform for selecting, customizing, and implementing virtual instruments for use with imaging devices (e.g., MRI, CT scan, X-ray, ultrasound, etc.) or sequencing devices at facility 3102. In at least one embodiment, virtual instruments can include software-defined applications for performing one or more processing operations with respect to imaging data generated by imaging devices, sequencing devices, radiology devices, and / or other device types. In at least one embodiment, one or more applications in a pipeline can use or call services of deployment system 3106 (e.g., inference, visualization, computation, AI, etc.) during execution of an application.

[0352] In at least one embodiment, some of the applications for advanced processing and inference pipelines can use machine learning models or other AI to perform one or more processing steps. In at least one embodiment, machine learning models can be trained using data 3108 (such as imaging data) generated at facility 3102 (and stored on one or more picture archiving and communication systems (PACS) servers at facility 3102), can be trained using imaging or sequencing data 3108 from another facility or facilities (e.g., different hospitals, labs, clinics, etc.), or a combination thereof. In at least one embodiment, training system 3104 can be used to provide applications, services, and / or other resources for generating work for deployment system 3106, deployable machine learning models.

[0353] In at least one embodiment, the model registry 3124 may be supported by an object storage capable of supporting versioning and object metadata. In at least one embodiment, the object storage may be, for example, cloud storage (e.g., FIG. 32 The cloud platform (3226)-compatible application programming interface (API) is accessed from within the cloud platform. In at least one embodiment, machine learning models in the model registry 3124 can be uploaded, listed, modified, or removed by the developer or partner of the system interacting with the API. In at least one embodiment, the API can provide access to methods that allow a user with appropriate credentials to associate a model with an application, enabling the model to be executed as part of the containerized instantiation of the application.

[0354] In at least one embodiment, training pipeline 3204 ( FIG. 32 This may include a scenario where facility 3102 is training its own machine learning model or has an existing machine learning model that needs to be optimized or updated. In at least one embodiment, imaging data 3108 generated by one or more imaging devices, sequencing devices, and / or other device types may be received. In at least one embodiment, once the imaging data 3108 is received, AI-assisted annotation 3110 may be used to help generate annotations corresponding to the imaging data 3108 for use as ground-based data for machine learning models. In at least one embodiment, AI-assisted annotation 3110 may include one or more machine learning models (e.g., convolutional neural networks (CNNs)) that may be trained to generate annotations corresponding to certain types of imaging data 3108 (e.g., from certain devices) and / or certain types of anomalies in the imaging data 3108. In at least one embodiment, AI-assisted annotation 3110 may then be used directly or may be adjusted or fine-tuned using annotation tools (e.g., by researchers, clinicians, doctors, scientists, etc.) to generate ground-based data. In at least one embodiment, in some examples, labeled clinical data 3112 (e.g., annotations provided by clinicians, doctors, scientists, technicians, etc.) can be used as ground-based data for training a machine learning model. In at least one embodiment, AI-assisted annotations 3110, labeled clinical data 3112, or a combination thereof can be used as ground-based data for training a machine learning model. In at least one embodiment, the trained machine learning model can be referred to as output model 3116 and can be used by deployment system 3106, as described herein.

[0355] In at least one embodiment, training pipeline 3204 ( FIG. 32) can include scenarios in which facility 3102 requires a machine learning model for performing one or more processing tasks for one or more applications in deployment system 3106, but facility 3102 can not currently have such a machine learning model (or can not have an optimized, efficient, or effective model for such purposes). In at least one embodiment, an existing machine learning model can be selected from model registry 3124. In at least one embodiment, model registry 3124 can include machine learning models trained to perform a variety of different inferencing tasks on imaging data. In at least one embodiment, machine learning models in model registry 3124 can have been trained on imaging data from a facility different from facility 3102 (e.g., a facility located remotely). In at least one embodiment, machine learning models can have been trained on imaging data from one location, two locations, or any number of locations. In at least one embodiment, when training on imaging data from a particular location, training can occur at that location, or at least in a manner that protects the confidentiality of the imaging data or limits the imaging data from being transmitted off-site (e.g., to comply with HIPAA regulations, privacy regulations, etc.). In at least one embodiment, once a model is trained or partially trained at a location, the machine learning model can be added to model registry 3124. In at least one embodiment, the machine learning model can then be retrained or updated at any number of other facilities, and the retrained or updated model can be made available in model registry 3124. In at least one embodiment, the machine learning model can then be selected from model registry 3124— and referred to as output model 3116— and can be used in deployment system 3106 to perform one or more processing tasks for one or more applications of the deployment system.

[0356] In at least one embodiment, training pipeline 3204( FIG. 32), a scenario can include a facility 3102 that requires a machine learning model for performing one or more processing tasks for one or more applications in a deployment system 3106, but the facility 3102 can not currently have such a machine learning model (or can not have an optimized, efficient, or effective model for such purposes). In at least one embodiment, due to population differences, a machine learning model selected from the model registry 3124 can not be fine-tuned or optimized for imaging data 3108 generated at the facility 3102, genetic variations, robustness of training data used to train the machine learning model, diversity of anomalies of the training data, and / or other issues with the training data. In at least one embodiment, AI assisted annotation 3110 can be used to help generate annotations corresponding to the imaging data 3108 that is used as ground truth data for retraining or updating the machine learning model. In at least one embodiment, labeled clinical data 3112 (e.g., annotations provided by a clinician, physician, scientist, etc.) can be used as ground truth data for training the machine learning model. In at least one embodiment, retraining or updating the machine learning model can be referred to as model training 3114. In at least one embodiment, the model training 3114 (e.g., AI assisted annotation 3110, labeled clinical data 3112, or a combination thereof) can be used as ground truth data for retraining or updating the machine learning model. In at least one embodiment, a trained machine learning model can be referred to as an output model 3116, and can be used by the deployment system 3106, as described herein.

[0357] In at least one embodiment, deployment system 3106 can include software 3118, services 3120, hardware 3122, and / or other components, features, and functionality. In at least one embodiment, deployment system 3106 can include a software “stack” such that software 3118 can be built on top of services 3120 and can use services 3120 to perform some or all processing tasks, and services 3120 and software 3118 can be built on top of hardware 3122 and use hardware 3122 to perform processing, storage, and / or other computing tasks of deployment system 3106. In at least one embodiment, software 3118 can include any number of different containers, where each container can execute an instantiation of an application. In at least one embodiment, each application can perform one or more processing tasks (e.g., inference, object detection, feature detection, segmentation, image enhancement, calibration, etc.) in a high-level processing and inference pipeline. In at least one embodiment, for each type of imaging device (e.g., CT, MRI, X-ray, ultrasound, echocardiogram, etc.), sequencing device, radiology device, genomics device, etc., there can be any number of containers that can perform data processing tasks with respect to imaging data 3108 (or other data types such as those described herein) generated by the device. In at least one embodiment, a high-level processing and inference pipeline can be defined based on a selection of different containers that are to or needed to process imaging data 3108, in addition to containers that receive and configure imaging data for use by each container and / or after processing through the pipeline for use by facility 3102 (e.g., convert output back to usable data types such as Digital Imaging and Communications in Medicine (DICOM) data, Radiology Information System (RIS) data, Clinical Information System (CIS) data, Remote Procedure Call (RPC) data, data substantially compliant with a Representational State Transfer (REST) interface, data substantially compliant with a file-based interface, and / or raw data for storage and display at facility 3102). In at least one embodiment, a combination of containers within software 3118 (e.g., a combination of containers that make up a pipeline) can be referred to as a virtual tool (as described in greater detail herein), and a virtual tool can utilize services 3120 and hardware 3122 to perform some or all processing tasks of applications instantiated in containers.

[0358] In at least one embodiment, a data processing pipeline can receive input data (e.g., imaging data 3108) in DICOM, RIS, CIS, REST compatible, RPC, raw, and / or other formats in response to an inference request (e.g., a request from a user (such as a clinician, physician, radiologist, etc.) of deployment system 3106). In at least one embodiment, input data can represent one or more images, videos, and / or other data representations generated by one or more imaging devices, sequencing devices, radiology devices, genomics devices, and / or other device types. In at least one embodiment, data can undergo pre-processing as part of a data processing pipeline to prepare data for one or more applications. In at least one embodiment, post-processing can be performed on output of one or more inference tasks or other processing tasks of a pipeline in order to prepare output data for a next application and / or to prepare output data for transmission and / or use by a user (e.g., in response to an inference request). In at least one embodiment, inference tasks can be performed by one or more machine learning models, such as trained or deployed neural networks, which can include output models 3116 of training system 3104.

[0359] In at least one embodiment, tasks of a data processing pipeline can be encapsulated in one or more containers, which each represent a discrete, fully-functional instantiation of an application and virtualized computing environment capable of referencing a machine learning model. In at least one embodiment, containers or applications can be published into a private (e.g., restricted access) area of a container registry (described in greater detail herein), and trained or deployed models can be stored in model registry 3124 and associated with one or more applications. In at least one embodiment, images (e.g., container images) of applications can be available in a container registry, and once selected by a user from a container registry for deployment in a pipeline, an image can be used to generate a container for instantiation of an application for use by a user’s system.

[0360] In at least one embodiment, developers (e.g., software developers, clinicians, physicians, etc.) can develop, publish, and store applications (e.g., as containers) for performing image processing and / or inference on provided data. In at least one embodiment, development, publication, and / or storage can be performed using a software development kit (SDK) associated with a system (e.g., to ensure that developed applications and / or containers are compatible or compatible with a system). In at least one embodiment, at least some of services 3120 can be leveraged as a system (e.g., to provide a developer with a platform to develop, publish, and store applications and / or containers for use by a system). FIG. 32The system 3200's SDK is used to test the developed application locally (e.g., at the first facility, on data from the first facility). In at least one embodiment, because DICOM objects can include anywhere from one to hundreds of images or other data types, and due to variations in data, the developer can be responsible for managing (e.g., setting up construction, incorporating preprocessing into the application, etc.) the extraction and preparation of the input DICOM data. In at least one embodiment, once validated by the system 3200 (e.g., for accuracy, security, patient privacy, etc.), the application can be available in the container registry for users (e.g., hospitals, clinics, laboratories, healthcare providers, etc.) to select and / or implement to perform one or more processing tasks on data at the user's facility (e.g., a second facility).

[0361] In at least one embodiment, the developer can then share the application or container over the network for the system (e.g., FIG. 32 The system 3200 allows for user access and use. In at least one embodiment, completed and validated applications or containers may be stored in a container registry, and associated machine learning models may be stored in a model registry 3124. In at least one embodiment, a requesting entity (e.g., a user at a medical facility) providing an inference or image processing request can browse the container registry and / or model registry 3124 of applications, select desired combinations of elements such as containers, datasets, machine learning models, etc., to include in the data processing pipeline, and submit an imaging processing request. In at least one embodiment, the request may include input data necessary to perform the request (and, in some examples, associated patient data), and / or may include the selection of one or more applications and / or machine learning models to be performed in the processing request. In at least one embodiment, the request may then be passed to one or more components of the deployment system 3106 (e.g., the cloud) to perform processing in the data processing pipeline. In at least one embodiment, the processing performed by the deployment system 3106 may include referencing the selected elements (e.g., applications, containers, models, etc.) from the container registry and / or model registry 3124. In at least one embodiment, once results are generated by the pipeline, they can be returned to the user for reference (e.g., for viewing in a suite of viewing applications running locally, on a field workstation, or on a terminal). In at least one embodiment, radiologists can receive results from a data processing pipeline that includes any number of applications and / or containers, where results may include the detection of abnormalities in X-rays, CT scans, MRIs, etc.

[0362] In at least one embodiment, to help process or execute applications or containers in a pipeline, services 3120 can be utilized. In at least one embodiment, services 3120 can include compute services, artificial intelligence (AI) services, visualization services, and / or other service types. In at least one embodiment, services 3120 can provide functionality that is common to one or more applications in software 3118, and thus functionality can be abstracted to a service that can be called or utilized by applications. In at least one embodiment, functionality provided by services 3120 can be run dynamically and more efficiently, while also scaling well by allowing applications to process data in parallel (e.g., using parallel computing platform 3230 FIG. 32 ). In at least one embodiment, services 3120 can be shared between and among different applications, rather than requiring each application sharing the same functionality provided by services 3120 to have a respective instance of services 3120. In at least one embodiment, as a non-limiting example, a service can include an inference server or engine that can be used to perform detection or segmentation tasks. In at least one embodiment, a model training service can be included, which can provide machine learning model training and / or retraining capabilities. In at least one embodiment, a data augmentation service can be further included, which can provide GPU-accelerated data (e.g., DICOM, RIS, CIS, REST-compliant, RPC, raw, etc.) extraction, resizing, scaling, and / or other augmentations. In at least one embodiment, a visualization service can be used, which can add image rendering effects - such as ray tracing, rasterization, de-noising, sharpening, etc. - to add realism to two-dimensional (2D) and / or three-dimensional (3D) models. In at least one embodiment, a virtual instrument service can be included that provides beamforming, segmentation, inference, imaging, and / or support for other applications within a pipeline that are virtual instruments.

[0363] In at least one embodiment, where services 3120 include an AI service (e.g., an inference service), one or more machine learning models associated with an application for anomaly detection (e.g., tumors, growth anomalies, scarring, etc.) can be executed, or processing thereof, as part of application execution by calling (e.g., as an API call) an inference service (e.g., an inference server) to execute the one or more machine learning models. In at least one embodiment, where another application includes one or more machine learning models for a segmentation task, an application can call to an inference service to execute the machine learning models for performing one or more of the processing operations associated with the segmentation task. In at least one embodiment, software 3118 implementing a high-level processing and inference pipeline including a segmentation application and an anomaly detection application can be streamlined, as each application can call the same inference service to perform one or more inference tasks.

[0364] In at least one embodiment, hardware 3122 can include GPUs, CPUs, graphics cards, AI / deep learning systems (e.g., AI supercomputers such as NVIDIA’s DGX), cloud platforms, or a combination thereof. In at least one embodiment, different types of hardware 3122 can be used to provide efficient, purpose-built support for software 3118 and services 3120 in deployment system 3106. In at least one embodiment, use of GPU processing can enable improved efficiency, image processing, image reconstruction, segmentation, accuracy and efficacy of MRI exams, stroke or heart attack detection (e.g., in real-time), image quality in rendering, etc. within AI / deep learning systems, in cloud systems, and / or in other processing components of deployment system 3106. In at least one embodiment, a facility can include imaging devices, genomics devices, sequencing devices, and / or other device types in the field that can utilize GPUs to generate imaging data representative of a subject’s anatomy. In at least one embodiment, software 3118 and / or services 3120 can be optimized for GPU processing with respect to deep learning, machine learning, and / or high performance computing, by way of non-limiting example. In at least one embodiment, at least some of a computing environment of deployment system 3106 and / or training system 3104 can execute one or more supercomputers or high performance computing systems in a data center with GPU-optimized software (e.g., a hardware and software combination of NVIDIA’s DGX systems). In at least one embodiment, a data center can comply with HIPAA regulations such that receipt, processing, and transmission of imaging data and / or other patient data is handled securely with respect to privacy of patient data. In at least one embodiment, hardware 3122 can include any number of GPUs that can be invoked to perform processing of data in parallel, as described herein. In at least one embodiment, a cloud platform can also include GPU processing for GPU-optimized execution of deep learning tasks, machine learning tasks, or other computing tasks. In at least one embodiment, an AI / deep learning supercomputer and / or GPU-optimized software (e.g., provided on NVIDIA’s DGX systems) can be used as a hardware abstraction and scaling platform to execute a cloud platform (e.g., NVIDIA’s NGC). In at least one embodiment, a cloud platform can integrate an application container clustering system or orchestration system (e.g., KUBERNETES) on multiple GPUs to enable seamless scaling and load balancing.

[0365] FIG. 32 is a system diagram of an example system 3200 for generating and deploying imaging deployment pipelines, in accordance with at least one embodiment. In at least one embodiment, system 3200 can be used to implement FIG. 31The process 3100 and / or other processes including advanced processing and inference pipelines. In at least one embodiment, system 3200 may include training system 3104 and deployment system 3106. In at least one embodiment, training system 3104 and deployment system 3106 may be implemented using software 3118, service 3120 and / or hardware 3122, as described herein.

[0366] In at least one embodiment, system 3200 (e.g., training system 3104 and / or deployment system 3106) may be implemented in a cloud computing environment (e.g., using cloud 3226). In at least one embodiment, system 3200 may be implemented locally relative to a healthcare service facility, or as a combination of cloud computing resources and local computing resources. In at least one embodiment, in the cloud-based implementation, patient data may be separated from or processed by one or more components of system 3200 that would render processing non-compliant with HIPAA and / or other data processing and privacy regulations or laws. In at least one embodiment, access to APIs in cloud 3226 may be restricted to authorized users through established security measures or protocols. In at least one embodiment, the security protocol may include a web token that can be signed by an authentication service (e.g., AuthN, AuthZ, Gluecon, etc.) and carries appropriate authorization. In at least one embodiment, the API of the virtual tool (described herein) or other instances of system 3200 may be restricted to a set of public IPs that have been reviewed or authorized for interaction.

[0367] In at least one embodiment, the various components of system 3200 can communicate with each other and with each other using any of a variety of different network types, including but not limited to local area networks (LANs) and / or wide area networks (WANs), via wired and / or wireless communication protocols. In at least one embodiment, communication between facilities and components of system 3200 (e.g., for transmitting inference requests, for receiving the results of inference requests, etc.) can be transmitted via a data bus, a wireless data protocol (Wi-Fi), a wired data protocol (e.g., Ethernet), etc.

[0368] In at least one embodiment, the training system 3104 may execute the training pipeline 3204, similar to that described herein. FIG. 31The described embodiments. In at least one embodiment, wherein one or more machine learning models will be used in deployment pipeline 3210 by deployment system 3106, training pipeline 3204 may be used to train or retrain one or more (e.g., pre-trained) models, and / or implement one or more pre-trained models 3206 (e.g., without retraining or updating). In at least one embodiment, one or more output models 3116 may be generated as a result of training pipeline 3204. In at least one embodiment, training pipeline 3204 may include any number of processing steps, such as, but not limited to, imaging data (or other input data) conversion or adaptation (e.g., converting DICOM images to another format suitable for processing by the corresponding machine learning model, such as the Neuroimaging Information Technology Inspired (NIfTI) format) using DICOM adapter 3202A, AI-assisted annotation 3110, labeling or annotating imaging data 3108 to generate labeled clinical data 3112, model selection from model registry, model training 3114, training, retraining or updating the model, and / or other processing steps. In at least one embodiment, different training pipelines 3204 can be used for different machine learning models used by the deployment system 3106. In at least one embodiment, similar to see [reference needed]. FIG. 31 The training pipeline 3204 described in the first example can be used for the first machine learning model, similar to see [link to example]. FIG. 31 The training pipeline 3204 described in the second example can be used for a second machine learning model, and is similar to that described above. FIG. 31 The training pipeline 3204 of the third example described can be used for a third machine learning model. In at least one embodiment, any combination of tasks within the training system 3104 can be used depending on what each corresponding machine learning model needs. In at least one embodiment, one or more machine learning models may have already been trained and are ready for deployment, so the machine learning models may not undergo any processing performed by the training system 3104 and can be implemented by the deployment system 3106.

[0369] In at least one embodiment, depending on the implementation or embodiment, one or more output models 3116 and / or one or more pre-trained models 3206 may include any type of machine learning model. In at least one embodiment, but not limited to, the machine learning model used by system 3200 may include one or more machine learning models using linear regression, logistic regression, decision trees, support vector machines (SVM), primitive Bayes, k-nearest neighbors (Knn), K-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoders, convolutions, recursion, perceptrons, long / short-term memory (LSTM), Hopfield, Boltzmann, deep belief, deconvolution, adversarial generation, liquid machines, etc.) and / or other types of machine learning models.

[0370] In at least one embodiment, training pipeline 3204 may include AI-assisted annotations, as described herein at least regarding FIG. 33BDescribed in more detail. In at least one embodiment, labeled clinical data 3112 can be generated using any number of techniques (e.g., conventional annotation). In at least one embodiment, labels or other annotations can be generated within a drawing program (e.g., annotating program), a computer-aided design (CAD) program, a labeling program, another type of program suitable for generating annotations or labels for ground reality, and / or in some examples, can be hand-drawn. In at least one embodiment, ground reality data can be synthetically generated (e.g., generated from a computer model or rendering), realistically generated (e.g., designed and generated from real-world data), machine-automated (e.g., extracting features from data using feature analysis and learning and then generating labels), manually annotated (e.g., labelers or annotation experts defining the placement of labels), and / or a combination thereof. In at least one embodiment, for each example of imaging data 3108 (or other data types used by a machine learning model), there may be corresponding ground reality data generated by training system 3104. In at least one embodiment, AI-assisted annotation can be performed as part of deployment pipeline 3210; as a supplement to or alternative to AI-assisted annotation included in training pipeline 3204. In at least one embodiment, system 3200 may include a multi-layer platform that may include a software layer (e.g., software 3118) capable of performing one or more medical imaging and diagnostic functions for diagnostic applications (or other application types). In at least one embodiment, system 3200 may be communicatively coupled to (e.g., via an encrypted link) a network of PACS servers in one or more facilities. In at least one embodiment, system 3200 may be configured to access and reference data (e.g., DICOM data, RIS data, raw data, CIS data, REST-compatible data, RPC data, raw data, etc.) from a PACS server (e.g., via DICOM adapter 3202 or another data type adapter such as RIS, CIS, REST-compatible data, RPC data, raw data, etc.) to perform operations such as training machine learning models, deploying machine learning models, image processing, inference, and / or other operations.

[0371] In at least one embodiment, the software layer may be implemented as a secure, encrypted, and / or certified API through which an application or container can be invoked (e.g., called) from one or more external environments (e.g., facility 3102). In at least one embodiment, the application may then invoke or execute one or more services 3120 to perform computational, AI, or visualization tasks associated with the respective application, and the software 3118 and / or service 3120 may utilize the hardware 3122 to perform processing tasks in an efficient and effective manner.

[0372] In at least one embodiment, deployment system 3106 may execute deployment pipeline 3210. In at least one embodiment, deployment pipeline 3210 may include any number of applications that can be applied sequentially, non-sequentially, or otherwise to imaging data (and / or other data types) generated by imaging devices, sequencing devices, genomics devices, etc.—including AI-assisted annotation as described above. In at least one embodiment, as described herein, deployment pipeline 3210 for an individual device may be referred to as a virtual instrument for the device (e.g., a virtual ultrasound instrument, a virtual CT scanner, a virtual sequencing instrument, etc.). In at least one embodiment, for a single device, there may be more than one deployment pipeline 3210 depending on the information required from the data generated by the device. In at least one embodiment, a first deployment pipeline 3210 may exist when it is desired to detect an abnormality from an MRI machine, and a second deployment pipeline 3210 may exist when it is desired to perform image enhancement from the output of the MRI machine.

[0373] In at least one embodiment, the applications available for deployment pipeline 3210 may include any application that can perform processing tasks on imaging data or other data from the device. In at least one embodiment, different applications may be responsible for image enhancement, segmentation, reconstruction, anomaly detection, object detection, feature detection, treatment planning, dosimetry, beam planning (or other radiotherapy procedures), and / or other analysis, image processing, or inference tasks. In at least one embodiment, deployment system 3106 may define a construction for each application such that users of deployment system 3106 (e.g., medical facilities, laboratories, clinics, etc.) can understand the construction and adapt the application for implementation within their respective facilities. In at least one embodiment, an application for image reconstruction may be selected to be included in deployment pipeline 3210, but the data type generated by the imaging device may differ from the data type used within the application. In at least one embodiment, a DICOM adapter 3202B (and / or a DICOM reader) or another data type adapter or reader (e.g., RIS, CIS, REST-compatible, RPC, raw, etc.) may be used within deployment pipeline 3210 to convert data into a form usable by applications within deployment system 3106. In at least one embodiment, access to DICOM, RIS, CIS, REST-compatible, RPC, raw, and / or other data type libraries can be accumulated and preprocessed, including decoding, extraction, and / or performing any convolution, color correction, sharpening, gamma, and / or other enhancements to the data. In at least one embodiment, DICOM, RIS, CIS, REST-compatible, RPC, and / or raw data can be unordered, and pre-passing can be performed to organize or sort the collected data. In at least one embodiment, because different applications can share common image operations, in some embodiments, data enhancement libraries (e.g., as one of services 3120) can be used to accelerate these operations. In at least one embodiment, to avoid bottlenecks in conventional processing methods that rely on CPU processing, a parallel computing platform 3230 can be used for GPU acceleration of these processing tasks.

[0374] In at least one embodiment, the image reconstruction application may include processing tasks that include using a machine learning model. In at least one embodiment, a user may expect to use their own machine learning model or select a machine learning model from a model registry 3124. In at least one embodiment, a user may implement their own machine learning model or select a machine learning model to include in the application used to perform the processing task. In at least one embodiment, the application may be optional and customizable, and by defining the construction of the application, the deployment and implementation of the application for a particular user is presented as a more seamless user experience. In at least one embodiment, by leveraging other features of system 3200—such as service 3120 and hardware 3122—the deployment pipeline 3210 can be even more user-friendly, providing easier integration and producing more accurate, efficient, and timely results.

[0375] In at least one embodiment, deployment system 3106 may include user interface 3214 (e.g., graphical user interface, web interface, etc.) which may be used to select applications to be included in deployment pipeline 3210, schedule applications during setup and / or deployment, modify or change applications or their parameters or configurations, use deployment pipeline 3210 and interact with deployment pipeline 3210, and / or otherwise interact with deployment system 3106. In at least one embodiment, although not shown relative to training system 3104, user interface 3214 (or different user interfaces) may be used to select models to be used in deployment system 3106, to select models to be used for training or retraining in training system 3104, and / or to otherwise interact with training system 3104.

[0376] In at least one embodiment, in addition to the application coordination system 3228, a pipeline manager 3212 may be used to manage interactions between applications or containers deployed through the pipeline 3210 and services 3120 and / or hardware 3122. In at least one embodiment, the pipeline manager 3212 may be configured to facilitate interactions from application to application, from application to service 3120, and / or from application or service to hardware 3122. In at least one embodiment, although shown as included in software 3118, this is not intended to be limiting, and in some examples, the pipeline manager 3212 may be included in service 3120. In at least one embodiment, the application coordination system 3228 (e.g., Kubernetes, DOCKER, etc.) may include a container orchestration system that can group applications into containers as logical units for coordination, management, scaling, and deployment. In at least one embodiment, by associating applications from the deployment pipeline 3210 (e.g., refactored applications, segmented applications, etc.) with separate containers, each application can execute in a self-contained environment (e.g., at the kernel level) to improve speed and efficiency.

[0377] In at least one embodiment, each application and / or container (or an image thereof) can be developed, modified, and deployed independently (e.g., a first user or developer can develop, modify, and deploy a first application, while a second user or developer can develop, modify, and deploy a second application separate from the first user or developer). This allows focus on and attention to the tasks of individual applications and / or containers without being hindered by the tasks of other applications or containers. In at least one embodiment, communication and collaboration between different containers or applications can be facilitated by pipeline manager 3212 and application coordination system 3228. In at least one embodiment, as long as the system (e.g., based on the application or container's architecture) knows the expected inputs and / or outputs of each container or application, application coordination system 3228 and / or pipeline manager 3212 can facilitate communication between and between them, as well as resource sharing between and among each application or container. In at least one embodiment, because one or more applications or containers in deployment pipeline 3210 can share the same services and resources, application coordination system 3228 can coordinate, load balance, and determine the sharing of services or resources between and among different applications or containers. In at least one embodiment, the scheduler can be used to track the resource requirements of applications or containers, the current or planned use of these resources, and resource availability. In at least one embodiment, the scheduler can thus allocate resources to different applications and distribute resources among and between applications based on the system's needs and availability. In some examples, the scheduler (and / or other components of the application coordination system 3228) can determine resource availability and distribution based on constraints imposed on the system (e.g., user constraints), such as Quality of Service (QoS), the urgency of needing data output (e.g., determining whether to perform...

Claims

1. A processor, comprising: One or more circuits for using one or more neural networks to: -Remove a first object from the first image, wherein the first object is similar in type to a second object from the second image; - Perform restoration on a region of the first image that previously corresponded to the first object, wherein the restoration includes replacing the region with inferred scene content; and - Detect one or more anomalies in the repaired first image, and if the one or more anomalies are detected, regenerate the first image to attempt to remove the one or more anomalies.

2. The processor of claim 1, wherein the one or more neural networks comprise a plurality of variational autoencoders (VAEs) trained to encode image features of different categories of objects into a latent space.

3. The processor of claim 2, wherein the one or more neural networks further include a gating network for selecting one of the VAEs to encode image features of the second object of the type into the latent space.

4. The processor of claim 3, wherein the one or more neural networks further comprises a generative adversarial network (GAN) for generating an output image based on the image content of the first image, and using the latent space as a constraint such that the output image does not include image content corresponding to the first object, wherein the GAN performs inpainting on regions of the first image previously corresponding to the first object.

5. The processor of claim 1, wherein the one or more neural networks are further configured to detect one or more instances of the type of object in the first image after removing the first object, and to cause the first image to be regenerated in an attempt to remove the one or more instances.

6. A system comprising: One or more processors are used to: -Remove a first object from the first image, wherein the first object is similar in type to a second object from the second image; - Perform restoration on a region of the first image that previously corresponded to the first object, wherein the restoration includes replacing the region with inferred scene content; and - Detect one or more anomalies in the repaired first image, and if the one or more anomalies are detected, regenerate the first image to attempt to remove the one or more anomalies.

7. The system of claim 6, wherein the one or more neural networks comprise a plurality of variational autoencoders (VAEs) trained to encode image features of different categories of objects into a latent space.

8. The system of claim 7, wherein the one or more neural networks further include a gating network for selecting one of the VAEs to encode image features of the second object of the type into the latent space.

9. The system of claim 8, wherein the one or more neural networks further include a generative adversarial network (GAN) for generating an output image based on the image content of the first image, and using the latent space as a constraint such that the output image does not include image content corresponding to the first object, wherein the GAN performs inpainting on regions of the first image previously corresponding to the first object.

10. The system of claim 6, wherein the one or more neural networks are further configured to detect one or more instances of the type of object in the first image after the first object has been removed, and to cause the first image to be regenerated in an attempt to remove the one or more instances.

11. A method comprising: Use one or more neural networks to remove a first object from a first image, wherein the first object is similar in type to a second object from a second image; Repair is performed on a region of the first image that previously corresponded to the first object, wherein the repair includes replacing the region with inferred scene content; as well as Detect one or more anomalies in the repaired first image, and if the one or more anomalies are detected, regenerate the first image to attempt to remove the one or more anomalies.

12. The method of claim 11, wherein the one or more neural networks comprise a plurality of variational autoencoders (VAEs) trained to encode image features of different categories of objects into a latent space.

13. The method of claim 12, wherein the one or more neural networks further include a gating network for selecting one of the VAEs to encode image features of the second object of the type into the latent space.

14. The method of claim 13, wherein the one or more neural networks further comprise a generative adversarial network (GAN) for generating an output image based on image content of the first image, and using the latent space as a constraint such that the output image does not include image content corresponding to the first object, wherein the GAN performs inpainting on regions of the first image previously corresponding to the first object.

15. The method of claim 11, wherein the one or more neural networks are further configured to detect one or more instances of the type of object in the first image after removing the first object, and to regenerate the first image to attempt to remove the one or more instances.

16. A machine-readable medium having a set of instructions stored thereon, said instructions, if executed by one or more processors, causing said one or more processors to at least: Use one or more neural networks to remove a first object from a first image, wherein the first object is similar in type to a second object from a second image; Repair is performed on the region of the first image that previously corresponded to the first object, wherein, The repair includes replacing the area with inferred scene content; as well as Detect one or more anomalies in the repaired first image, and if the one or more anomalies are detected, regenerate the first image to attempt to remove the one or more anomalies.

17. The machine-readable medium of claim 16, wherein the one or more neural networks comprise a plurality of variational autoencoders (VAEs) trained to encode image features of different categories of objects into a latent space.

18. The machine-readable medium of claim 17, wherein the one or more neural networks further include a gating network for selecting one of the VAEs to encode image features of the second object of the type into the latent space.

19. The machine-readable medium of claim 18, wherein the one or more neural networks further comprise a generative adversarial network (GAN) for generating an output image based on image content of the first image, and using the latent space as a constraint such that the output image does not include image content corresponding to the first object, wherein the GAN performs inpainting on regions of the first image previously corresponding to the first object.

20. The machine-readable medium of claim 16, wherein the one or more neural networks are further configured to detect one or more instances of objects of the type in the first image after the first object has been removed, and to cause the first image to be regenerated in an attempt to remove the one or more instances.

21. An image generation system, comprising: One or more processors are used for: - Use one or more neural networks to remove a first object from a first image, wherein the first object is similar in type to a second object from a second image; Perform restoration on a region of the first image that previously corresponded to the first object, wherein the restoration includes replacing the region with inferred scene content; and Detect one or more anomalies in the repaired first image; if the one or more anomalies are detected, regenerate the first image to attempt to remove the one or more anomalies; and A memory for storing network parameters of the one or more neural networks.

22. The image generation system of claim 21, wherein the one or more neural networks comprise a plurality of variational autoencoders (VAEs) trained to encode image features of different categories of objects into a latent space.

23. The image generation system of claim 22, wherein the one or more neural networks further include a gating network for selecting one of the VAEs to encode image features of the second object of the type into the latent space.

24. The image generation system of claim 23, wherein the one or more neural networks further include a generative adversarial network (GAN) for generating an output image based on the image content of the first image, and using the latent space as a constraint such that the output image does not include image content corresponding to the first object, wherein the GAN performs inpainting on a region of the first image previously corresponding to the first object.

25. The image generation system of claim 21, wherein the one or more neural networks are further configured to detect one or more instances of the type of object in the first image after removing the first object, and to regenerate the first image to attempt to remove the one or more instances.

Images