Generation of super-resolution images using neural networks

Abstract

The approaches presented here provide systems and procedures for content generation systems that incorporate a Denoising Diffusion Generative Adversarial Network (DDGAN) into a content generation pipeline. A generator connected to the DDGAN can be conditioned on a set of input images containing at least one noisy image from a diffusion engine, one low-resolution upsampled image, and one historical image. The generator can be used to produce an output image that has one or more properties distinct from those of a content engine. The weights for the generator can be determined during a training procedure that includes a discriminator evaluating at least the noisy image, the low-resolution upsampled image, and a noisy image generated from the output image.

Description

BACKGROUND

[0001] The hardware capabilities of facilities can limit the resolution or other image quality factors for content generation systems. Users may want to consume content at a specific resolution or frame rate, among other preferences, but the associated facilities may lack the hardware capabilities to do so. Often, users choose between a variety of factors, such as image rendering, performance, and the like. When a user has to make these trade-offs, the overall quality of the user experience with a given product can be negatively impacted. Super-resolution models can be used to upsample an image from a content engine using one or more trained neural networks, but these models may be slow, unstable, or unable to achieve the desired resolution. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Various embodiments according to the present disclosure are described with reference to the drawings, in which: Fig. 1 An exemplary schematic representation of a super-resolution pipeline according to various embodiments is illustrated; Fig. Figure 2 illustrates an exemplary environment for training a generator in a super-resolution pipeline according to various embodiments; Fig. Figure 3 illustrates an exemplary method for determining a loss for the training of a generator in a pipeline according to different embodiments; Fig. Figure 4 illustrates an exemplary procedure for adjusting network parameters for a generator according to different embodiments; Fig. Figure 5 illustrates an exemplary method for generating an image from a content generation pipeline according to various embodiments; Fig. Figure 6 illustrates components of a distributed system that can be used to update or perform inferencing using a machine learning model according to at least one embodiment; Fig. 7A illustrates an inference and / or training logic according to at least one embodiment; Fig. 7B illustrates an inference and / or training logic according to at least one embodiment; Fig. Figure 8 illustrates an exemplary data center system according to at least one embodiment; Fig. Figure 9 illustrates a computer system according to at least one embodiment; Fig. 10 illustrates a computer system according to at least one embodiment; Fig. 11 illustrates at least sections of a graphics processor according to one or more embodiments; Fig. 12 illustrates at least sections of a graphics processor according to one or more embodiments; Fig. Figure 13 is an exemplary data flow diagram for an advanced computing pipeline according to at least one embodiment; Fig. Figure 14 is a system representation for an exemplary system for training, adapting, instantiating and deploying machine learning models in an advanced computing pipeline according to at least one embodiment; Fig. 15A and Fig. Figure 15B illustrates a data flow diagram for a method for training a machine learning model and a client-server architecture for improving annotation tools with pre-trained annotation models according to at least one embodiment; DETAILED DESCRIPTION

[0003] The following description details various embodiments. For explanatory purposes, specific configurations and details are presented to provide a thorough understanding of the embodiments. However, it is also apparent to a person skilled in the art that the embodiments can be implemented without these specific details. Furthermore, known features may be omitted or simplified so as not to obscure the described embodiment.

[0004] The systems and procedures described here can be used by, among others, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in an infotainment or digital application or a virtual driver assistant in the cabin), autonomous vehicles or machines, controlled and uncontrolled robots or robot platforms, warehouse vehicles, off-road vehicles, vehicles coupled with one or more trailers, aircraft, boats, shuttles, emergency vehicles, motorcycles, electric or motorized bicycles, airplanes, construction vehicles, trains, underwater vehicles, remotely controlled vehicles such as drones and / or other types of vehicles.Furthermore, the systems and methods described here can be used for a wide variety of purposes, including but not limited to machine control, machine locomotion, machine driving, synthetic data generation, model training or updating, perception, augmented reality, virtual reality, mixed reality, robotics, security and monitoring, simulation and digital twin technology, applications for autonomous or semi-autonomous machines, deep learning, environmental simulation, object or actuator simulation and / or a digital twin, data center processing, conversational artificial intelligence (AI), generative AI with large language models (LLMs) and / or visual language models (VLMs), and light transport simulation (e.g., ray tracing, path tracing, etc.).), a collaborative creation of content for 3D objects, cloud computing and / or other suitable applications.

[0005] The disclosed embodiments can be included in a variety of different systems, such as automotive systems (e.g., a control system for an autonomous or semi-autonomous machine, a perception system for an autonomous or semi-autonomous machine), systems implemented using a robot, flight systems, media systems, boat systems, intelligent area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented with an edge device, systems containing one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in a data center, and systems for performing conversational AI operations.Systems for performing generative AI operations using LLMs, systems for performing light transport simulations, systems for performing collaborative content creation for 3D assets, systems that are implemented at least partially using cloud computing resources, and / or other types of systems.

[0006] Approaches according to various embodiments are geared towards a temporal super-resolution model that combines principles of Deep Learning Super Sampling (DLSS) and Denoising Diffusion Generative Adversarial Networks (GANs) (DDGANs). Systems and methods adapt a DDGAN architecture featuring forward diffusion and post-sampling to provide additional inputs to both a generator and a discriminator. In at least one embodiment, the DDGAN includes a reference image, which may be an image generated by downsampling a high-resolution image, and performs forward diffusion to generate a pair of noisy images. Each of these noisy images can be used as input for one or both, the discriminator and the generator. Furthermore, concepts from DLSS can also be applied to generate an image obtained by upscaling or downsampling.Upsampling of a low-resolution image and a historical image (which, depending on the image being processed, can be a previously generated image or the upsampled low-resolution image) are used as additional inputs for both the generator and the discriminator. Accordingly, the generator can be used to produce an output image based on each of the noisy image, the upsampled low-resolution image, and the historical image. This output image is then processed by the discriminator to adjust the weights for both the discriminator and the generator during a training phase. Combining the two architectures addresses the inference speed and instability issues of both methods to provide an improved super-resolution system.

[0007] Several embodiments of the present disclosure can overcome problems associated with super-resolution models. Typically, models involve trade-offs between sample quality, inference speed, and mode coverage / training stability. For example, diffusion models can produce high-quality images with high mode coverage and stable training, but their inference speeds can be slow, making them unsuitable or suboptimal for certain applications, such as those requiring more than single-image generation, including video playback. GANs can also be used for image generation and can be trained to produce high-quality images with faster inference, but with lower training stability.Other models may utilize architectures such as U-Nets, which are fast but have limited image quality. In contrast, one or more embodiments of the present disclosure combine the advantages of deep learning super-resolution systems and GANs to generate high-quality images quickly and with high stability. Accordingly, various embodiments can provide systems that increase performance while managing memory requirements and upfront training costs.

[0008] In at least one embodiment, systems and methods reorient a temporal super-resolution architecture by conditioning an input to a generator and discriminator with a low-resolution input. Additionally, a history buffer can be used to provide an extra input in the form of a most recently generated image. Unlike general diffusion models, these systems and methods allow the output to be computed during the training time for the next image, as the inclusion of the DDGAN enables more cost-effective evaluation compared to diffusion models. The low-resolution input and the history buffer can also help the network maintain plausibility during upsampling and stability across images. Accordingly, various embodiments can be used to enhance content generation pipelines, such as…those that involve generating multiple images or videos.

[0009] Within the framework of the various embodiments, various other such functions can also be used, as would be obvious to a person skilled in the art in this field in view of the teachings and proposals contained herein.

[0010] Fig. Figure 1 shows an exemplary environment 100 that can be used with embodiments of the present disclosure. The depicted environment 100 has a super-resolution pipeline 102 that takes as input one or more images generated by a machine 104, such as a game or video machine, and produces an output 106 that may contain one or more images. Certain embodiments may describe the input as an “image,” and it should be noted that the image may be a standalone single image or an image extracted from a video sequence, among other options. The super-resolution pipeline 102 may be integrated into a general rendering or development pipeline, such as one connected to a content generation system running on a computing device. For example, the engine 104 may be a game engine that renders a video (e.g.,a series of images) rendered in conjunction with a video game or other type of content. As noted herein, video games are given as a non-limiting example, and various other applications may include the Super Resolution Pipeline 102 and / or features of the Super Resolution Pipeline 102. In certain embodiments, the Engine 104 may operate based on one or more user-provided settings that adjust various parameters of the Engine 104. The settings and / or parameters may be linked to output parameters for the engine, such as resolution, frame rate, anti-aliasing, and / or the like. For example, the Engine 104 may include settings for interacting with the content at different resolutions, frame rates, texture qualities, colors, and / or the like.In a non-restrictive example, which serves only as an illustration, resolution can refer to the number of pixels on a screen and be associated with perceived image quality. That is, a "higher" resolution can produce a better quality image than a "lower" resolution. Quality can be measured using subjective or objective methods. Subjective measurements might be based, for example, on the perception of a user evaluating the image. In contrast, objective measurements can involve one or more mathematical or statistical procedures, including various full-reference, reduced-reference, or no-reference methods. Examples of objective measurement methods include mean squared error, maximum signal-to-noise ratio, structural similarity, visual information fidelity, and the like.

[0011] In at least one embodiment, the associated hardware may limit or otherwise restrict various output settings. To return to the resolution example: A user with a 4K Ultra High Definition (UHD) monitor may be able to view images at 4K UHD (e.g., at least 3840 pixels by 2160 pixels), while a user with a 1080p monitor may only be able to view images at lower resolutions, e.g., 1920 pixels by 1080 pixels. Similarly, the hardware capabilities of the computing device may limit or restrict the resolution of the resulting output images, for example, by using a GPU (Graphics Processing Unit) that lacks sufficient processing capabilities or by using "on-board" processing (e.g., a combined CPU (Central Processing Unit) without dedicated graphics).The systems and methods of this disclosure can include the super-resolution pipeline 102 to generate higher-quality images (e.g., images with a perceived higher quality due to their higher resolution), while reducing memory and power consumption by executing one or more trained neural networks to generate high-resolution images (e.g., with a resolution exceeding a threshold) with a resolution greater than that of a reference image generated by the engine 104. The systems and methods can also be used to enhance other visual aspects of various content generation pipelines, which may include generating and inserting new images into an image sequence to achieve a higher frame rate.In this way, Engine 104 settings can be optimized for properties other than resolution or frame rate, while still producing high-quality output images (e.g., images that have "higher" settings than the Engine 104 output settings).

[0012] In this example, the super-resolution pipeline 102 includes a resolution model 108 (e.g., a super-resolution model capable of upsampling a low-resolution image to a high resolution) and a DDGAN model 110 (e.g., a DDGAN). During the operation, input images are produced or generated by the engine 104 and, after undergoing one or more preprocessing operations in certain embodiments, can be provided to both the resolution model 108 and the DDGAN 110 to produce output images with a higher resolution than those generated by the engine 104. For example, an input image generated by the engine 104 can be passed to the resolution model 108 to produce an upsampled image (e.g., a low-resolution image undergoing upsampling).The resolution model 108 can be designed for fast execution and can be a lightweight, low-latency model. Engine 104 can generate an image (e.g., a single image or a series of images in a video sequence) at an initial resolution and provide the image(s) to the resolution model 108. Using one or more trained networks, the resolution model 108 then analyzes the image(s) (e.g., a frame from a video sequence or content sequence) and creates a new image that may have a higher resolution than the original input image(s). Various implementations can employ supersampling to sample the lower-resolution input image and then use the samples to construct a higher-resolution output image.In this way, engine 104 can be set to produce images with a lower resolution than the final output image produced by resolution model 108.

[0013] One or more embodiments can employ a DDGAN that is not modeled by a Gaussian distribution, but instead minimizes the divergence between diffuse real-world data distributions and a diffuse generator distribution over multiple time steps. During training, a generator can be updated by backpropagating the gradient of a forward diffusion chain and then by a discriminator. Furthermore, various embodiments can feed a diffusion-based Gaussian mixture distribution into the discriminator, allowing the denoising process to approximate multiple steps simultaneously. Accordingly, various embodiments can enable faster sample generation, where samples are generated from noise using the DDGAN, providing an improved pipeline that addresses and overcomes latency issues in legacy systems.DDGAN systems and methods can model the denoising distribution using a complex multimodal distribution. During operation, a noisy observation can be fed to the DDGAN and / or its components, and then an attempt is made to generate a less noisy sample. The GAN can be trained using adversarial losses. Once an image is generated, a discriminator based on the same noisy observation can attempt to distinguish whether a denoised sample was generated by the generator or acquired in the previous step of the forward process (e.g., sampled from a series of images generated by the diffusion model).

[0014] Systems and procedures can also integrate the DDGAN 110 into the super-resolution pipeline 102. The DDGAN 110 can incorporate a forward diffusion method, also known as Gaussian diffusion, to extract samples from a pair of diffuse samples at different time steps. For example, an initial input image, such as a reference image generated by the engine 104, can be provided as input for the forward diffusion method to generate one or more intermediate images containing noise. Different levels of noise can be added at different times, with "earlier" steps having less noise than "later" steps, to give just one example. The forward diffusion can continue across a denoising distribution. A discriminator can also be part of the DDGAN 110 to receive one of the generated intermediate images as input, as well as a final noisy image.Furthermore, a generator can receive the noisy image as input to provide a generated output sample. For example, the generator can be connected to a conditional denoising GAN that stochastically generates images. Post-processing sampling can also be performed, and then the generated image can be provided to the discriminator, for example, during a training procedure to determine whether the generated image is genuine or fake. The weights can then be adjusted for the generator and the discriminator.

[0015] The systems and methods of the present disclosure can combine the DDGAN 110 with additional inputs from the resolution model 108 to generate an output image with reduced memory consumption and reduced latency. For example, various embodiments can provide the generator and / or the discriminator with the low-resolution upsampled image generated by the DLSS, along with one or more historical images or single frames. In this way, the generator and the discriminator can be conditioned to additional inputs and historical data, thereby reducing training runs and generating higher-quality output images.

[0016] The super-resolution pipeline 102 can also include a rendering engine 112 to obtain the output of the DDGAN 110, and in certain embodiments, the resolution model 108 and / or the engine 104, generating output 106, which may be in the form of one or more images. Accordingly, systems and methods can integrate the super-resolution pipeline 102 into an image or graphics rendering pipeline to generate output images with higher resolutions and / or frame rates, including those generated by the engine 104.

[0017] Fig. Figure 2 shows an exemplary training environment 200 that can be used with embodiments of the present disclosure. In this example, a training environment can be created using a DDGAN, such as the DDGAN 110 from Fig. 1. The connected architecture can be modified to provide one or more different inputs to a discriminator and / or generator to adapt different training weights for the discriminator and / or generator. In this example, the DDGAN architecture includes a diffusion engine 202, a generator 204, a discriminator 206, and a post-sampling engine 208. In various embodiments, the DDGAN's GAN can receive a noisy observation and then attempt to generate a less noisy sample. The GAN is trained through an adversarial loss, and the discriminator 206, trained on the same noisy observation, can attempt to distinguish whether a denoised sample was generated by the generator 204 or acquired in a previous step (e.g., the forward diffusion step).During sampling, the process can begin with white noise and repeat the denoising step multiple times to reduce the white noise to clean data. Accordingly, embodiments of DDGAN 110 introduce a generative model as a combination of diffusion models and GANs, with each denoising step of the diffusion model being modeled with a conditional GAN.

[0018] The diffusion engine 202 of the illustrated embodiment can receive a reference image 210 as input and perform a forward diffusion over a distribution (q(x) t-1|x0)) is applied to gradually add noise to the reference image 210 over a number of time steps. This process can be called "forward diffusion," where noise is added gradually, so that earlier time steps may have less noise than later time steps. It should be noted that in certain embodiments, a reverse process can also be performed to remove the noise from the reference image 210. In this example, the reference image 210 may correspond to a high-resolution image that has undergone downsampling. For example, one or more preprocessing steps may be applied to the reference image 210 to reduce its resolution from an initial value to a lower value, such as reducing a 16K resolution image to 4K or 1080p. The initial reference image may be generated by an engine, such asA game engine generates the image and then downsamples it to achieve a specific target resolution. In operation, the diffusion engine 202 can be used to gradually add noise to the reference image 210 across the distribution. The noise can be Gaussian in certain embodiments, but it can also be non-Gaussian. Assuming Gaussian noise, the number of denoising steps can be very large, which supports the Gaussian assumption. However, if the denoising steps are large (e.g., a threshold value greater than that assumed for a Gaussian distribution), the denoising distribution may not be Gaussian. Therefore, systems and methods can attempt to reduce the number of denoising steps by applying a multimodal distribution.

[0019] In at least one embodiment, forward diffusion is set up with the assumption that a small number of denoising steps (T) is used (e.g., below a threshold used in current models). It should be noted that the "small" number of steps T can vary, at least in part, based on the application being processed by the system. In at least one embodiment, T is less than or equal to 8. However, it should be noted that T can also be less than or equal to 4. Furthermore, in at least one embodiment, T can be between 4 and 8. Additionally, in other embodiments, T can be greater than 8, greater than 10, greater than 15, greater than 50, or within various ranges such as between 5 and 10, between 10 and 15, between 10 and 30, between 50 and 100, or other reasonable ranges.

[0020] As mentioned previously, the diffusion engine 202 can feature forward and / or reverse processing. The forward process can gradually add noise to the reference image 210 up to a certain point (e.g., up to a point where the reference image 210 reaches a threshold, at which point the reference image may be rendered essentially unrecognizable). In the reverse process, the number of denoising steps can be reduced, as mentioned above, and a non-Gaussian, multimodal distribution can be used to model the denoising distribution. By eliminating the Gaussian assumption, reducing the number of steps allows the generation quality to be maintained while simultaneously improving speed. Accordingly, the diffusion engine 202 can be used to generate both an intermediate image 212 (x t-1 )as well as a noisy image 214 (x tto generate. The intermediate image 212 exhibits more noise than the reference image 210, which may be an essentially clear and / or higher-quality image. However, as noted here, the reference image 210 may have a lower resolution / be less clear than an engine-generated output image, for example, due to one or more preprocessing steps that may include downsampling. The forward diffusion can occur across the entire denoising distribution (q(x)). t |x t-1 )) up to the noisy image 214.

[0021] In the configuration shown, the intermediate image 212 and the noisy image 214 are provided as inputs for the discriminator 206. Furthermore, the noisy image 214 is then provided to the generator 204, which outputs a generated image 216 (x'0). The GAN can, for example, be a conditional denoising GAN that stochastically generates the generated image 216. Subsequently, the engine 208 is used for post-processing sampling to generate a noisy generated image 218 (x'). t-1 ) to generate. As shown, the generated image 216 is assigned a distribution (q(x') t-1 |x tNoise is added again to form the noisy generated image 118. The noisy generated image 118 can then be provided to the discriminator 206 along with the intermediate image 212 and the noisy image 214 among other inputs described here. As will be described later, in various embodiments, the intermediate image 212 and the noisy generated image 218 can be provided to the discriminator 206 without being present during the same pass. The conditional GAN ​​can then be trained to denoise inputs using an adversarial loss for different steps in the diffusion process. For example, an output 220 of the discriminator 206 can be used to determine whether certain images are "real" or "fake," and the weights 222 can be adjusted for the generator 204 and / or the discriminator 206, respectively.

[0022] The illustrated embodiment of Fig. Section 2 contains an additional latent variable 224 (z) as input for generator 206. The latent variable 224 can be used as z~N(O,I) Group normalization layers (GN layers) can be defined, and one or more group normalization layers (GN layers) in the network can be replaced by adaptive group normalization layers (AdaGN layers) to allow the input of latent variables. The latent variable 224 can be mapped by a fully connected network (referred to as a mapping network in certain embodiments), and then a resulting embedding vector is sent to each AdaGN layer. Each AdaGN layer contains a fully connected layer that takes the embedding vector as input and outputs the shift and scale parameters per channel for group normalization.

[0023] In at least one embodiment, the discriminator 206 is a time-dependent discrimination designed with a convolutional network of ResNet blocks, which may be similar to that of the generator 204. The discrimination 206 is positioned to discriminate the intermediate image 212, which is dependent on the noisy image 214 and time. The time conditioning can be enforced by the sinusoidal position embeddings. Furthermore, the condition of the noisy image 214 can be enforced by concatenating the noisy image 214 and the intermediate image 212 as input to the discriminator 206. As noted herein, embodiments may also provide one or more additional inputs for the discriminator 206.

[0024] In at least one embodiment, the intermediate image 212 is provided to both the discriminator 206 and the generator 204 after diffusion in the diffusion engine 202. The discriminator 206 can then evaluate one or more input images to provide feedback for improving the generator 204, for example by adjusting one or more weights 222. After training, the generator 204 can then provide an output based on one or more input images.

[0025] Various embodiments of the present disclosure may further include a low-resolution upsampled image 226 and a historical image 228 as inputs for conditioning the generator 204 and for evaluation by the discriminator 206. In at least one embodiment, the low-resolution upsampled image 226 may be generated by an engine, such as a game engine, at a lower resolution than the resolution "set" or desired for image output, but which has been subjected to one or more upsampling processes, such as processing using one or more resolution engines, as described herein. For example, the engine may be configured to generate images at 540p and then upsample them to 1080p.In at least one embodiment, both the reference image 210 and the upsampled low-resolution image 226 can have the same resolution.

[0026] The historical image 228 shown can be acquired from one or more buffers during system training. On the first pass, historical image 228 may be the same as the upsampled, low-resolution image 226. Subsequently, historical image 228 may be the generated image 216 from a previous pass, among various other options.

[0027] As shown, various embodiments can be used to modify a DDGAN architecture to condition generator 204 to the noisy image 214, the upsampled, low-resolution image 226, and the historical image 228. By conditioning generator 204 to the additional images (e.g., more than just the noisy image 214 or typical DDGANs), the systems and procedures can reduce the probability of training collapse. For example, it may be preferable to have a generator that is "best" for its given discriminator rather than a "near-perfect" generator or a "near-perfect" discriminator.This means that if the generator produces images at the beginning of its training that the discriminator cannot distinguish, the generator should not be further adjusted or trained. This is not only due to the quality of the generator, but possibly also to a weakness in the discriminator. Another example: If the discriminator is too adept at distinguishing between real and fake images, the generator will not have enough training opportunities to improve.

[0028] In at least one embodiment, this can be achieved in Fig. The training methods described in Figure 2 can be performed over several iterations for each reference. For example, different embodiments can "traverse" the sequence four times, to give just one non-restrictive example. Accordingly, in at least one embodiment, on a first pass, the historical image 228 can correspond to the upsampled, low-resolution image 226. Then, on the second pass, the historical image 228 can correspond to the generated image 216 produced after the first pass. Then, on the third pass, the historical image 228 can correspond to the generated image 216 produced after the second pass, and so on. This iterative process can be repeated for each image in a sequence, such as a video sequence or the output of a computer game engine. Once a predetermined number of passes has been completed (e.g.,(four as a non-restrictive example), the next image can be selected as the reference image 210 and the procedure can be repeated.

[0029] Various embodiments can implement the trained generator 204 in a super-resolution pipeline, where output images from the generator 204 can be used to upscale the resolution of an image (e.g., an image from a video sequence, a generated content element, etc.) and / or to generate additional images to increase the frame rate or other visual parameter of an input sequence. Using a computer game engine as an example, the engine can output a series of frames (e.g., images) at a specific frame rate. An initial image from the series can be provided as a reference image 210, and further images can be generated, such as the upsampled, lower-resolution image 226. For the first pass, the historical image 228 can correspond to the upsampled, lower-resolution image 226.A first iteration can be performed in which generator 204 produces the generated image 216. The generated image 216 associated with the first iteration can then be used as the historical image 228 for the second iteration, continuing to use the same reference image 210 and the same low-resolution upsampled image 226 as initial inputs. The process can be repeated a certain number of times until a final generated image 216 is produced, which can then be provided for rendering, for example, as part of a sequence to increase the frame rate and / or resolution, among other things. Then, the next image in the sequence can be provided as the reference image 210, and the process can be repeated. In at least one embodiment, every image in the sequence is processed, but in other embodiments, specific images can be selected, such as...every second picture or something like that.

[0030] In this example, the inputs provided to the discriminator 206 can include the noisy image 214, the upsampled low-resolution image 226, the historical image 228, the intermediate image 212, and / or the noisy generated image 218. The selected image can be based on the specific discriminator iteration performed during the training procedure. At each training step, the discriminator 206 processes both, but in different embodiments, not in the same pass.

[0031] Various implementations can offer improvements to existing rendering pipelines by allowing the original engine that generates the input content to run at a lower resolution and / or frame rate, among other things, and to produce higher quality output images by consuming fewer processing resources and / or less memory.

[0032] Fig. Figure 3 shows an exemplary flowchart for an exemplary method 300 for training a content generation system that can be used with embodiments of the present disclosure. It should be understood that for this and other methods presented herein, additional, fewer, or alternative operations may be performed in a similar or alternative sequence, or at least partially in parallel, within different embodiments, unless expressly stated otherwise. In this example, a first and a second noisy image are generated from a reference image using forward diffusion 302. For example, one or more embodiments may include a diffusion engine that is used to take an input image, apply noise over a series of time points, and then generate different noisy images as a result of incorporating the noise.An upsampled image can also be generated based on the reference image 304. As noted herein, the upsampled image can be generated using one or more pipelines, such as a resolution engine, which receives output from an engine or source with a first resolution and then generates another image or frame with a second resolution, which may differ from the first resolution, such as a higher resolution. In at least one embodiment, the upsampled image represents a scene from the reference image.

[0033] Various embodiments can also be used to generate an output image, for example, by using a generator as part of a GAN 306. For example, the generator can receive one or more inputs, including the upsampled image, the second noisy image, and a historical image, to condition the generator to produce the output image. In at least one embodiment, additional noise can also be fed into the generator. A noisy output image can then be generated based on at least the output image 308, for example, based on post-sampling. One or more of the images can then be used to determine a loss parameter for the generator 310.For example, a discriminator can be used to compare the second noisy image, the upsampled image, the historical image, and the first noisy image and / or the noisy output image. In this way, the loss parameter can be used to adjust or change the weights associated with the generator and / or the discriminator to fine-tune or adjust subsequent output images from the generator.

[0034] Fig. Figure 4 shows an exemplary flowchart for an exemplary method 400 for setting parameters of a generator that can be used with embodiments of the present disclosure. In this example, a series of noisy images is generated from a reference image 402. For example, a reference image can be provided as input to a diffusion engine that incrementally adds noise to a series of images over a distribution. A final noisy image can be selected from the series of noisy images 404 and, together with a low-resolution upsampled image and a historical image 406, fed to a generator to produce an output image. In at least one embodiment, a noisy output image is generated using an output image produced by the generator 408.A comparison can then be made between the final noisy image, the upsampled low-resolution image, the historical image, and the noisy output image and / or the noisy intermediate image from the series of noisy images 410. This comparison can be used to adjust generator parameters, such as a loss function, and to modify subsequent output images of the generator.

[0035] Fig. Figure 5 shows an exemplary flowchart for an exemplary method 500 for generating content within a content generation and production pipeline that can be used with embodiments of the present disclosure. In this example, a reference image is obtained 502. The reference image can be generated by a content generation engine, such as a game engine, which can be part of the pipeline. In at least one embodiment, a low-resolution image corresponding to the reference image can also be obtained 504. The low-resolution image can have a lower resolution than the desired final output resolution, but embodiments can generate an upsampled low-resolution image 506. The upsampled low-resolution image can have a higher resolution than the low-resolution image.

[0036] In certain embodiments, a generative pipeline can be used to generate one or more output images and can be executed iteratively over a predetermined number of steps. It can be determined which step of the process is associated with the current reference image 508. For a first step, a historical image can be defined that corresponds to the upsampled, low-resolution image 510. However, if previous steps have already been executed, the historical image can be equated with a previous output image of the generator 512. A noisy image can be generated from the reference image using a diffusion mesh and provided to a trained generator 514. Additional inputs to the generator can include the upsampled, low-resolution image and / or the historical image. The generator can then be used to produce an output image 516.As mentioned previously, a single reference image can go through several steps to produce a final output image. It can be determined whether further steps are pending (518), and if not, the output image can be provided as the final output image (520). This image can be fed into a content generation pipeline to produce output content with different resolutions than the original input content.

[0037] As discussed, aspects of various approaches presented here can be lightweight or simple enough to be executed in real time on a client device, such as a personal computer or game console. Such processing can be performed on or for content created on or received by that client device, or received from an external source, such as streaming data or other content received over at least one network. In some cases, the processing and / or determination of this content can be performed by one of these other devices, systems, or entities and then made available to the client device (or other recipient) for presentation or other use.

[0038] As an example, Fig. 6 An exemplary network configuration 600 that can be used to provide, generate, modify, encode, process, and / or transmit image data or other such content. In at least one embodiment, a client device 602 can generate or receive data for a session using components of a control application 604 on the client device 602 and data stored locally on that client device. In at least one embodiment, a content application 624 running on a server 620 (e.g., a cloud server or edge server) can initiate a session connected to at least one client device 602, using a session manager and user data stored in a user database 636, and can cause content such as one or more digital assets (e.g.,Object representations) from an asset repository 634 are determined by a content manager 626. A content manager 626 can cooperate with an image synthesis module 628 to generate or synthesize new objects, digital assets, or other such content, which are made available for presentation via the client facility 602. In at least one embodiment, this image synthesis module 628 can use one or more neural networks or machine learning models that can be trained or updated by means of a training module 632 or a system that resides on or communicates with the server 620.This may involve training and / or using a diffusion model 630 to generate content tiles that can be used by an image synthesis module 628 to, for example, apply a non-repeating texture to an area of ​​an environment for which image or video data is to be presented via the client device 602. At least a portion of the generated content can be transferred to the client device 602 by using a suitable transfer manager 622 to send it via download, streaming, or another such transfer channel. An encoder can be used to encode and / or compress at least some of this data before transferring it to the client device 602.In at least one embodiment, the client device 602, which receives such content, can make this content available to a corresponding control application 604, which may also or alternatively include a graphical user interface 610, a content manager 612, and an image synthesis or diffusion module 614 for use in providing, synthesizing, modifying, or using content for presentation (or other purposes) on or by the client device 602. A decoder can also be used to decode data received over the network(s) 640 for presentation via the client device 602, e.g., image or video content via a display 606 and audio content, e.g., sounds and music, via at least one audio playback device 608, e.g., loudspeakers or headphones.In at least one embodiment, at least part of this content may already be stored, rendered, or accessible on the client device 602, so that transmission over the network 640 is not necessary, at least for this portion of the content, for example, if this content has been previously downloaded or stored locally on a hard disk or optical disk. In at least one embodiment, a transmission mechanism such as data streaming may be used to transfer this content from the server 620 or the user database 636 to the client device 602. In at least one embodiment, at least one portion of this content may be obtained, enhanced, and / or streamed from another source, such as a third-party service 660 or another client device 650, which may also contain a content application 662 for generating, enhancing, or providing content.In at least one embodiment, sections of this functionality can be executed using multiple computing units or multiple processors within one or more computing units, which may, for example, include a combination of CPUs and GPUs.

[0039] In this example, these client devices can include any suitable computing equipment, such as a desktop computer, a notebook, a set-top box, a streaming device, a game console, a smartphone, a tablet computer, a VR headset, AR glasses, a portable computer, or a smart TV. Each client device can submit a request over at least one wired or wireless network, such as the internet, Ethernet, a local area network (LAN), or a cellular network. In this example, these requests can be submitted to an address associated with a cloud provider that operates or controls one or more electronic resources in a cloud provider environment, such as a data center or server farm.In at least one embodiment, the request can be received or processed by at least one edge server located at a network edge and outside at least one security layer connected to the cloud provider's environment. This reduces latency by enabling client devices to interact with servers located closer to the server, while simultaneously improving the security of resources within the cloud provider's environment.

[0040] In at least one embodiment, such a system can be used to perform graphical operations. In other embodiments, such a system can be used for other purposes, for example, to provide image or video content to test or validate autonomous machine applications, or to perform deep learning operations. In at least one embodiment, such a system can be implemented with an edge facility or comprise one or more virtual machines (VMs). In at least one embodiment, such a system can be implemented at least partially in a data center or at least partially using cloud computing resources. INFERENCE AND TRAINING LOGIC

[0041] Fig. Figure 7A illustrates inference and / or training logic 715, which is used to perform inference and / or training operations associated with one or more embodiments. Details regarding the inference and / or training logic 715 are given below in conjunction with Fig. 7A and / or 7B provided.

[0042] In at least one embodiment, the inference and / or training logic 715 can, without limitation, include code and / or data memory 701 to store forward and / or output weighting and / or input / output data and / or other parameters for configuring neurons or layers of a neural network that is trained in aspects of one or more embodiments and / or used for inference. In at least one embodiment, the training logic 715 can include or be coupled to code and / or data memory 701 to store graph code or other software for controlling the timing and / or order in which weighting and / or other parameter information is loaded to configure logic that includes integer and / or floating-point units (collectively, arithmetic logic units (ALUs)).In at least one embodiment, a code, such as a graph code, loads weighting or other parameter information based on the architecture of a neural network to which the code corresponds into the processor ALU. In at least one embodiment, the code and / or data memory 701 stores the weighting parameters and / or input / output data of each layer of a neural network that is trained or used in conjunction with one or more embodiments during the forward propagation of input / output data and / or weighting parameters during training and / or inference using aspects of one or more embodiments. In at least one embodiment, any portion of the code and / or data memory 701 may be included in another on-chip or off-chip data memory, including the L1, L2, or L3 cache or system memory of a processor.

[0043] In at least one embodiment, any section of the code and / or data memory 701 can be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data memory 701 can be a cache memory, a dynamic randomly addressable memory (“DRAM”), a static randomly addressable memory (“SRAM”), a non-volatile memory (e.g., flash memory), or another type of memory.In at least one embodiment, the choice of whether the code and / or data storage 701 is, for example, internal or external to a processor, or comprises DRAM, SRAM, Flash or another type of memory, may depend on the available chip-internal versus chip-external memory, latency requirements of executed training and / or inference functions, batch size of data used in inferencing and / or training a neural network, or a combination of these factors.

[0044] In at least one embodiment, the inference and / or training logic 715 may, without limitation, include a code and / or data store 705 to store backward and / or output weights and / or input / output data corresponding to neurons or layers of a neural network that is trained and / or used for inference in aspects of one or more embodiments. In at least one embodiment, the code and / or data store 705 stores weighting parameters and / or input / output data of each layer of a neural network that is trained or used in conjunction with one or more embodiments during backward propagation of input / output data and / or weighting parameters during training and / or inference using aspects of one or more embodiments.In at least one embodiment, the training logic 715 can include or be coupled to a code and / or data memory 705 to store graph code or other software to control the timing and / or order in which weighting and / or other parameter information is loaded, and to configure logic that includes integer and / or floating-point units (collectively, arithmetic logic units (ALUs)). In at least one embodiment, a code, such as a graph code, loads weighting or other parameter information into the processor ALU based on the architecture of a neural network to which the code corresponds. In at least one embodiment, any portion of the code and / or data memory 705 can be contained in another on-chip or off-chip data memory, including the L1, L2, or L3 cache or system memory of a processor.In at least one embodiment, any section of the code and / or data memory 705 can be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data memory 705 can be a cache memory, a DRAM, an SRAM, a non-volatile memory (e.g., flash memory), or another type of memory. In at least one embodiment, the choice of whether the code and / or data memory 705 is, for example, internal or external to a processor, or consists of a DRAM, an SRAM, a flash memory, or some other type of memory, can depend on the available on-chip memory compared to the available off-chip memory, the latency requirements of training and / or inference functions performed, the batch size of data used in inferencing and / or training a neural network, or a combination of these factors.

[0045] In at least one embodiment, the code and / or data memory 701 and the code and / or data memory 705 can be separate memory structures. In at least one embodiment, the code and / or data memory 701 and the code and / or data memory 705 can be the same memory structure. In at least one embodiment, the code and / or data memory 701 and the code and / or data memory 705 can be partly the same memory structure and partly separate memory structures. In at least one embodiment, any part of the code and / or data memory 701 and the code and / or data memory 705 can be contained in another chip-internal or chip-external data memory, which includes an L1, L2, or L3 cache or system memory of a processor.

[0046] In at least one embodiment, the inference and / or training logic 715 may, without limitation, include one or more arithmetic logic units (“ALU(s)”) 710, including integer and / or floating-point units, to perform logical and / or mathematical operations that are based at least in part on or specified by training and / or inference code (e.g., graph code), wherein a result thereof may produce activations (e.g., output values ​​of layers or neurons within a neural network) that are stored in an activation memory 720 and are functions of input / output and / or weighting parameter data that are stored in the code and / or data memory 705 and / or the code and / or data memory 701.In at least one embodiment, activations stored in the activation memory 720 are generated according to algebraic and / or matrix-based mathematics performed by the ALU(s) 710 in response to the execution of instructions or other code, wherein weight values ​​stored in the code and / or data memory 705 and / or the code and / or data memory 701 are used as operands together with other values, such as systematic error values, gradient information, dynamic values, or other parameters or hyperparameters, one or all of which may be stored in the code and / or data memory 701 or the code and / or data memory 705 or in another on-chip or off-chip memory.

[0047] In at least one embodiment, the ALU(s) 710 are contained within one or more processors or other hardware logic devices or circuits, whereas in another embodiment, the ALU(s) 710 may be external to a processor or other hardware logic device or circuit that uses them (e.g., a coprocessor). In at least one embodiment, the ALU(s) 710 may be contained within the execution units of a processor or otherwise within a bank of ALUs that the execution units of a processor can access, either within the same processor or distributed across different processors of different types (e.g., central processing units, graphics processing units, fixed function units, etc.).In at least one embodiment, the code and / or data memory 701, the code and / or data memory 705, and the activation memory 720 can be located on the same processor or other hardware logic device or circuit, whereas in another embodiment, they can be located on different processors or other hardware logic devices or circuits, or a combination of identical and different processors or other hardware logic devices or circuits. In at least one embodiment, any portion of the activation memory 720 can be included in another on-chip or off-chip data memory, including the L1, L2, or L3 cache or system memory of a processor.Furthermore, inference and / or training code can be stored with other code that a processor or other hardware logic or circuitry can access and be retrieved and / or processed using the retrieval, decoding, scheduling, execution, shutdown, and / or other logical circuitry of a processor.

[0048] In at least one embodiment, the activation memory 720 can be a cache memory, a DRAM, an SRAM, a non-volatile memory (e.g., flash memory), or another type of memory. In at least one embodiment, the activation memory 720 can be located wholly or partially inside or outside one or more processors or other logic circuits. In at least one embodiment, the choice of whether the activation memory 720 is, for example, internal or external to a processor, or whether it consists of a DRAM, an SRAM, flash memory, or another type of memory, can depend on the available on-chip memory compared to the available external memory, the latency requirements of training and / or inference functions performed, the batch size of data used in inferencing and / or training a neural network, or a combination of these factors.In at least one embodiment, the in . Fig. 7A illustrated inference and / or training logic 715 can be used in conjunction with an application-specific integrated circuit (“ASIC”), such as a Google Tensorflow® Processing Unit, a Graphcore™ inference processing unit (IPU), or an Intel Corp. Nervana®-type processor (e.g., “Lake Crest”). In at least one embodiment, the Fig. Figure 7A illustrates inference and / or training logic 715 used in conjunction with central processing unit (CPU) hardware, graphics processing unit (GPU) hardware, or other hardware such as field programmable gate arrays (FPGAs).

[0049] Fig. Figure 7B illustrates inference and / or training logic 715 according to at least one or more embodiments. In at least one embodiment, the inference and / or training logic 715 may, without limitation, include hardware logic in which computing resources are dedicated or otherwise used exclusively in conjunction with weight values ​​or other information corresponding to one or more layers of neurons within a neural network. In at least one embodiment, the Fig. 7B illustrated inference and / or training logic 715 can be used in conjunction with an application-specific integrated circuit (ASIC), such as Google's Tensorflow® processing unit, a Graphcore™ inference processing unit (IPU), or an Intel Corp. Nervana® processor (e.g., "Lake Crest"). In at least one embodiment, the inference and / or training logic illustrated in Figure 7B can be used in conjunction with an application-specific integrated circuit (ASIC), such as Google's Tensorflow® processing unit, a Graphcore™ inference processing unit (IPU), or an Intel Corp. Nervana® processor (e.g., "Lake Crest"). In at least one embodiment, the logic illustrated in Figure 7B can be used in conjunction with an application-specific integrated circuit (ASIC), such as Google's Tensorflow® processing unit, a Graphcore™ inference processing unit (IPU), or an Intel Corp. Nervana® processor (e.g., "Lake Crest"). Fig. Figure 7B illustrates inference and / or training logic 715 used in conjunction with central processing unit (CPU) hardware, graphics processing unit (GPU) hardware, or other hardware, such as field-programmable gate arrays (FPGAs). In at least one embodiment, the inference and / or training logic 715 includes, without limitation, code and / or data memory 701 and code and / or data memory 705, which can be used to store code (e.g., graph code), weight values, and / or other information, including distortion values, gradient information, instantaneous values, and / or other parameter or hyperparameter information. In at least one embodiment, which is illustrated in Fig. As illustrated in Figure 7B, each of the code and / or data memory 701 and the code and / or data memory 705 is assigned to a dedicated computing resource, such as the computing hardware 702 and the computing hardware 706, respectively. In at least one embodiment, each of the computing hardware 702 and the computing hardware 706 comprises one or more ALUs that perform mathematical functions, such as linear algebraic functions, only on information stored in the code and / or data memory 701 and the code and / or data memory 705, respectively, with the result being stored in the activation memory 720.

[0050] In at least one embodiment, each of the code and / or data storage units 701 and 705 and the corresponding computing hardware 702 and 706 corresponds to different layers of a neural network, such that the resulting activation from a "memory / computing pair 701 / 702" of the code and / or data storage unit 701 and the computing hardware 702 is provided as input for the "memory / computing pair 705 / 706" of the code and / or data storage unit 705 and the computing hardware 706, in order to reflect the conceptual organization of a neural network. In at least one embodiment, each of the memory / computing pairs 701 / 702 and 705 / 706 can correspond to more than one layer of a neural network. In at least one embodiment, additional memory / computing pairs (not shown) may be included after or in parallel to the memory / computing pairs 701 / 702 and 705 / 706 in the inference and / or training logic 715. DATA CENTER

[0051] Fig. Figure 8 illustrates an exemplary data center 800, in which at least one embodiment can be used. In at least one embodiment, the data center 800 includes a data center infrastructure layer 810, a framework layer 820, a software layer 830, and an application layer 840.

[0052] In at least one embodiment, as in Fig. As shown in Figure 8, the data center infrastructure layer 810 can include a resource orchestrator 812, clustered compute resources 814, and node compute resources (“node RR”) 816(1) to 816(N), where “N” is any positive integer. In at least one embodiment, the node RR 816(1)-816(N) can include, but are not limited to, any number of central processing units (“CPUs”) or other processors (including accelerators, field-programmable gate arrays (“FPGAs”), graphics processing units, etc.), memory devices (e.g., dynamic read-only memory), data storage devices (e.g., solid-state or disk drives), network input / output devices (“NW I / O” devices), network switches, virtual machines (“VMs”), power modules, and cooling modules, etc.In at least one embodiment, one or more of the Node-RRs of Node-RR 816(1)-816(N) can be a server comprising one or more of the aforementioned computing resources.

[0053] In at least one embodiment, the grouped compute resources 814 can include separate groupings of node RRs housed in one or more racks (not shown), or many racks housed in data centers at different geographic locations (also not shown). Separate groupings of node RRs within grouped compute resources 814 can include grouped compute, network, storage, or memory resources that can be configured or allocated to support one or more workloads. In at least one embodiment, multiple node RRs, including CPUs or processors, can be grouped in one or more racks to provide compute resources to support one or more workloads.In at least one embodiment, one or more racks can also include any number of power modules, cooling modules and network switches in any combination.

[0054] In at least one embodiment, the resource orchestrator 812 can configure or otherwise control one or more node RR 816(1) to 816(N) and / or grouped computing resources 814. In at least one embodiment, the resource orchestrator 812 can include a management unit of a software design infrastructure (“SDI”) for the data center 800. In at least one embodiment, the resource orchestrator 812 can include hardware, software, or a combination thereof.

[0055] In at least one embodiment, as in Fig. As shown in Figure 8, the framework layer 820 includes a task scheduler 822, a configuration manager 824, a resource manager 826, and a distributed file system 828. In at least one embodiment, the framework layer 820 can include a framework to support software 832 of software layer 830 and / or one or more applications 842 of application layer 840. In at least one embodiment, the software 832 or the application(s) 842 can each be web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. In at least one embodiment, the framework layer 820 can, without restriction, be a type of web application framework for free and open-source software, such as Apache Spark™ (hereinafter “Spark”), which can use the distributed file system 828 for large-scale data processing (e.g., “Big Data”).In at least one embodiment, the task scheduler 822 can include a Spark driver to facilitate the scheduling of workloads supported by various layers of the data center 800. In at least one embodiment, the configuration manager 824 can be able to configure various layers, such as the software layer 830 and the framework layer 820, including Spark and the distributed file system 828, to support large-scale data processing. In at least one embodiment, the resource manager 826 can be able to manage clustered or grouped compute resources allocated or assigned to support the distributed file system 828 and the task scheduler 822. In at least one embodiment, the clustered or grouped compute resources can include a grouped compute resource 814 on the data center infrastructure layer 810.In at least one embodiment, the resource manager 826 can coordinate with the resource orchestrator 812 to manage these allocated or assigned computing resources.

[0056] In at least one embodiment, the software 832, which is included in software layer 830, may include software that is used by at least parts of node RR 816(1) to 816(N), grouped computing resources 814, and / or the distributed file system 828 of framework layer 820. One or more types of software may include, but are not limited to, web browsing software, email scanning software, database software, and streaming video content software.

[0057] In at least one embodiment, the application(s) 842 included in the application layer 840 may include one or more types of applications used by at least parts of node RR 816(1) to 816(N), grouped compute resources 814, and / or the distributed file system 828 of the framework layer 820. One or more types of applications may include any number of a genomics application, a cognitive compute application, and a machine learning application, including, but not limited to, training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), or other machine learning applications used in conjunction with one or more embodiments.

[0058] In at least one embodiment, the configuration manager 824, the resource manager 826, and the resource orchestrator 812 can implement any number and type of self-modifying actions based on any amount and type of data acquired in any technically feasible manner. In at least one embodiment, self-modifying actions can relieve a data center operator of the data center 800 from potentially making poor configuration decisions and potentially avoiding underutilized and / or underperforming sections of a data center.

[0059] In at least one embodiment, the Data Center 800 may include tools, services, software, or other resources for training one or more machine learning models, or for predicting or deriving information using one or more machine learning models according to one or more embodiments described herein. For example, in at least one embodiment, a machine learning model may be trained by calculating weighting parameters according to a neural network architecture using software and computing resources previously described with reference to the Data Center 800.In at least one embodiment, trained machine learning models corresponding to one or more neural networks can be used to infer or predict information using resources previously described in relation to the Computing Center 800 and weighting parameters calculated using one or more training techniques described in this document.

[0060] In at least one embodiment, the data center can use CPUs, application-specific integrated circuits (ASICs), GPUs, FPGAs, or other hardware to perform training and / or inference using the resources described above. Furthermore, one or more of the software and / or hardware resources described above can be configured as a service to allow users to train or perform information inference, such as image recognition, speech recognition, or other artificial intelligence services.

[0061] The inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. Details regarding the inference and / or training logic 715 are provided below in conjunction with Fig. 7A and / or 7B are provided. In at least one embodiment, the inference and / or training logic 715 can be provided in the system from Fig. 8 for inference or prediction operations, at least partially based on weighting parameters calculated using neural network training operations, neural network functions and / or neural network architectures or neural network use cases described in this document.

[0062] Such components can be used for content generation pipelines. COMPUTER SYSTEMS

[0063] Fig. Figure 9 is a block diagram illustrating an exemplary computer system, which may be a system with interconnected devices and components, a system-on-a-chip (SoC), or a certain combination thereof, and is formed with a processor that may include execution units for carrying out an instruction, according to at least one embodiment. In at least one embodiment, the computer system may, without limitation, include a component, such as a processor, to employ execution units that include logic for carrying out algorithms on process data, according to the present disclosure, as in the embodiment described in this document.In at least one embodiment, the Computer System 900 may include processors such as the PENTIUM® processor family, Xeon™, Itanium®, XScale™ and / or StrongARM™, Intel® Core™ or Intel® Nervana™ microprocessors available from Intel Corporation in Santa Clara, California, although other systems (including PCs with other microprocessors, technical workstations, set-top boxes, and the like) may also be used. In at least one embodiment, the Computer System 900 may run a version of the WINDOWS operating system available from Microsoft Corporation in Redmond, Washington, although other operating systems (for example, UNIX and Linux), embedded software, and / or graphical user interfaces may also be used.

[0064] The embodiments can be used on other devices, such as handheld devices and embedded applications. Some examples of handheld devices are mobile phones, Internet Protocol devices, digital cameras, personal digital assistants (PDAs), and portable PCs. In at least one embodiment, embedded applications can include a microcontroller, a digital signal processor (DSP), a system-on-a-chip, network computers (NetPCs), set-top boxes, network hubs, wide-area network switches (WANs), or any other system capable of executing one or more instructions according to at least one embodiment.

[0065] In at least one embodiment, the computer system 900 can, without limitation, include a processor 902, which can, without limitation, include one or more execution units 908 for training and / or inferring a machine learning model according to the techniques described in this document. In at least one embodiment, the computer system 900 is a single-processor desktop or server system, but in another embodiment, the computer system 900 can be a multi-processor system.In at least one embodiment, the processor 902 can, without restriction, include a complex instruction set computing (“CISC”) microprocessor, a reduced instruction set computing (“RISC”) microprocessor, a very long instruction word (“VLIW”) microprocessor, a processor implementing a combination of instruction sets, or any other processing device, such as a digital signal processor. In at least one embodiment, the processor 902 can be coupled to a processor bus 910, which can transmit data signals between the processor 902 and other components in the computer system 900.

[0066] In at least one embodiment, the processor 902 can, without limitation, include a Level 1 (“L1”) internal cache memory (“cache”) 904. In at least one embodiment, the processor 902 can have a single internal cache or multiple levels of internal cache. In at least one embodiment, the cache memory can be located outside the processor 902. Other embodiments, depending on the specific implementation and requirements, can also include a combination of both internal and external caches. In at least one embodiment, the register file 906 can store various types of data in various registers, including, without limitation, integer registers, floating-point registers, status registers, and instruction pointer registers.

[0067] In at least one embodiment, the execution unit 908, which includes without limitation logic for performing integer and floating-point operations, is also located in the processor 902. In at least one embodiment, the processor 902 may also include a microcode ("uCode") read-only memory (ROM) that stores microcode for certain macro instructions. In at least one embodiment, the execution unit(s) 908 may include logic for handling a packed instruction set 909. In at least one embodiment, by including a packed instruction set 909 in an instruction set of a general-purpose processor 902, together with associated circuitry for executing instructions, operations used by numerous multimedia applications can be performed using compressed data in a general-purpose processor 902.In one or more embodiments, numerous multimedia applications can be accelerated and executed more efficiently by using the full width of a processor's data bus to perform operations on compressed data, thereby potentially eliminating the need to transfer smaller data units across the processor's data bus to perform one or more operations on only one data element at a time.

[0068] In at least one embodiment, the execution unit 908 can also be used in microcontrollers, embedded processors, graphics devices, DSPs, and other types of logic circuits. In at least one embodiment, the computer system 900 can include a memory 920 without restriction. In at least one embodiment, the memory 920 can be implemented as a dynamic random access memory (“DRAM”) device, a static random access memory (“SRAM”) device, a flash memory device, or another type of memory device. In at least one embodiment, the memory 920 can store instructions 919 and / or data 921 represented by data signals that can be executed by the processor 902.

[0069] In at least one embodiment, a system logic chip can be coupled to the processor bus 910 and the memory 920. In at least one embodiment, the system logic chip can include, without restriction, a memory controller hub (“MCH”) 916, and the processor 902 can communicate with the MCH 916 via the processor bus 910. In at least one embodiment, the MCH 916 can provide a high-bandwidth memory path 918 for the memory 920 for storing instructions and data and for storing graphics instructions, data, and textures. In at least one embodiment, the MCH 916 can route data signals between the processor 902, the memory 920, and other components in the computer system 900 and bridge data signals between the processor bus 910, the memory 920, and a system I / O 922. In at least one embodiment, the system logic chip can provide a graphics port for coupling to a graphics controller.In at least one embodiment, the MCH 916 can be coupled to the memory 920 via a high-bandwidth memory path 918, and a graphics / video card 912 can be coupled to the MCH 916 via an Accelerated Graphics Port (“AGP”) interconnection 914.

[0070] In at least one embodiment, the computer system 900 can use the system I / O 922, which is a proprietary node interface bus, to couple the MCH 916 to the I / O control hub (“ICH”) 930. In at least one embodiment, the ICH 930 can provide direct connections to some I / O devices via a local I / O bus. In at least one embodiment, the local I / O bus can, without restriction, include a high-speed I / O bus for connecting peripheral devices to the memory 920, the chipset, and the processor 902. Examples may include, without limitation, an audio control 929, a firmware hub (“Flash BIOS”) 928, a wireless transceiver 926, a data storage device 924, a legacy I / O control 923 containing user input and keyboard interfaces 925, a serial expansion port 927, such as a universal serial bus (“USB”) port, and a network control 934.The 924 data storage device can include a hard disk drive, a floppy disk drive, a CD-ROM device, a flash memory device, or another mass storage device.

[0071] Illustrated in at least one embodiment Fig. 9 a system that includes interconnected hardware devices or “chips”, whereas in other embodiments Fig. 9 illustrates an exemplary system on a chip (“SoC”). In at least one embodiment, the devices can be interconnected using proprietary interconnects, standardized interconnects (e.g., PCIe), or a combination thereof. In at least one embodiment, one or more components of the computer system 900 are interconnected using Compute Express Link (CXL) interconnects.

[0072] The inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. Details regarding the inference and / or training logic 715 are provided below in conjunction with Fig. 7A and / or 7B are provided. In at least one embodiment, the inference and / or training logic 715 can be provided in the system from Fig. 9 for inference or prediction operations at least partially based on weighting parameters used with the neural network training operations, neural network functions and / or architectures or neural network use cases described herein.

[0073] Such components can be used for content generation pipelines.

[0074] Fig. Figure 10 is a block diagram illustrating an electronic device 1000 for using a processor 1010 according to at least one embodiment. In at least one embodiment, the electronic device 1000 can be, for example, and without limitation, a notebook, a tower server, a rack server, a blade server, a laptop, a desktop computer, a tablet, a mobile device, a telephone, an embedded computer, or any other suitable electronic device.

[0075] In at least one embodiment, the electronic device 1000 can, without limitation, include the processor 1010, which is communicatively coupled to any suitable number or type of components, peripherals, modules, or devices. In at least one embodiment, the processor 1010 is coupled using a bus or interface, such as a 1°C bus, a system management bus (“SMBus”), a low-pin-count (LPC) bus, a serial peripheral interface (“SPI”), a high-definition audio (“HDA”) bus, a serial advance technology attachment (“SATA”) bus, a universal serial bus (“USB”) (version 1, 2, 3), or a universal asynchronous receiver / transmitter (“UART”) bus. In at least one embodiment, the following is illustrated: Fig. 10 a system comprising interconnected hardware devices or “chips”, whereas in other embodiments Fig. 10. An exemplary system on a chip (“SoC”) can be illustrated. In at least one embodiment, the Fig. The devices illustrated in 10 are interconnected using proprietary interconnects, standardized interconnects (e.g., PCIe), or a certain combination thereof. In at least one embodiment, one or more components are made of Fig. 10 interconnected using Compute Express Link (CXL) interconnections.

[0076] In at least one embodiment, Fig. 10 a display 1024, a touchscreen 1025, a touchpad 1030, a near field communications unit (NFC) 1045, a sensor hub 1040, a thermal sensor 1046, an Express chipset (EC) 1035, a trusted platform module (TPM) 1038, a BIOS / firmware / flash memory (BIOS, FW flash) 1022, a DSP 1060, a drive 1020, such as a solid state drive (SSD) or a hard disk drive (HDD), a wireless local area network (WLAN) 1050, a Bluetooth unit 1052, a wireless wide area network (WWAN) 1056, a global Positioning system (global positioning system - GPS) 1055, a camera (“USB 3.0 camera”) 1054, such as a USB 3.The device may include a 0-camera and / or a low-power double-data-rate ("LPDDR") storage unit ("LPDDR3") 1015, implemented, for example, in an LPDDR3 standard. These components may each be implemented in any suitable manner.

[0077] In at least one embodiment, other components can be communicatively coupled to the processor 1010 via previously discussed components. In at least one embodiment, an accelerometer 1041, an ambient light sensor (“ALS”) 1042, a compass 1043, and a gyroscope 1044 can be communicatively coupled to the sensor hub 1040. In at least one embodiment, the thermal sensor 1039, a fan 1037, a keyboard 1036, and a touchpad 1030 can be communicatively coupled to the EC 1035. In at least one embodiment, one or more loudspeakers 1063, headphones 1064, and a microphone (“Mic”) 1065 can be communicatively coupled to an audio unit (“Class D audio codec and amplifier”) 1062, which in turn can be communicatively coupled to the DSP 1060. In at least one embodiment, the audio unit 1062 can, for example and without limitation, include an audio encoder / decoder (“codec”) and a class-D amplifier.In at least one embodiment, a SIM card (“SIM”) 1057 can be communicatively coupled with the WWAN unit 1056. In at least one embodiment, components such as the WLAN unit 1050 and the Bluetooth unit 1052, as well as the WWAN unit 1056, can be implemented in a next-generation form factor (“NGFF”).

[0078] The inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. Details regarding the inference and / or training logic 715 are provided below in conjunction with Fig. 7A and / or 7B are provided. In at least one embodiment, the inference and / or training logic 715 can be provided in the system from Fig. 10 for inference or prediction operations at least partly based on weighting parameters calculated using neural network training operations, neural network functions and / or neural network architectures or neural network use cases described in this document.

[0079] Such components can be used for content generation pipelines.

[0080] Fig. Figure 11 is a block diagram of a processing system according to at least one embodiment. In at least one embodiment, the system 1100 includes one or more processors 1102 and one or more graphics processors 1108 and can be a single-processor desktop system, a multi-processor workstation system, or a server system comprising a large number of processors 1102 or processor cores 1107. In at least one embodiment, the processing system 1100 is a processing platform integrated into an integrated circuit as a system-on-a-chip (“SoC”) for use in mobile, portable, or embedded devices.

[0081] In at least one embodiment, the System 1100 can comprise or be integrated into a server-based gaming platform, a game console comprising a game and media console, a mobile gaming console, a handheld game console, or an online game console. In at least one embodiment, the System 1100 is a mobile phone, a smartphone, a tablet computing device, or a mobile internet device. In at least one embodiment, the Processing System 1100 can also include, be coupled to, or be integrated into a wearable device, such as a smartwatch wearable device, a smart eyewear device, an augmented reality device, or a virtual reality device.In at least one embodiment, the processing system 1100 is a television or set-top box device comprising one or more processor(s) 1102 and a graphical user interface generated by one or more graphics processor(s) 1108.

[0082] In at least one embodiment, one or more processor(s) 1102 each include one or more processor core(s) 1107 for processing instructions that, when executed, perform operations for system and user software. In at least one embodiment, each of the processor core(s) 1107 is configured to process a specific instruction set 1109. In at least one embodiment, the instruction set 1109 can enable Complex Instruction Set Computing (CISC), Reduced Instruction Set Computing (RISC), or computation using a Very Long Instruction Word (VLIW). In at least one embodiment, the processor core(s) 1107 can each process a different instruction set 1109, which may contain instructions to facilitate the emulation of other instruction sets. In at least one embodiment, the processor core(s) 1107 can...The processor core(s) 1107 also include other processing devices, such as a digital signal processor (DSP).

[0083] In at least one embodiment, the processor(s) 1102 include a cache memory 1104. In at least one embodiment, the processor(s) 1102 may have a single internal cache or multiple levels of internal cache. In at least one embodiment, the cache memory is shared by different components of the processor(s) 1102. In at least one embodiment, the processor(s) 1102 also use an external cache (e.g., a Level 3 ("L3") cache or Last-Level Cache ("LLC")) (not shown), which can be shared by the processor core(s) 1107 using known cache coherence techniques. In at least one embodiment, the processor(s) 1102 additionally includes a register file 1106, which may contain different types of registers for storing different types of data (e.g.,(Integer register, floating-point register, status register, and an instruction pointer register). In at least one embodiment, the register file 1106 may contain universal registers or other registers.

[0084] In at least one embodiment, one or more processors 1102 are coupled to one or more interface buses 1110 to transmit communication signals, such as address, data, or control signals, between the processor 1102 and other components in the system 1100. In at least one embodiment, the interface bus(s) 1110 can be a processor bus, such as a version of a Direct Media Interface (DMI) bus. In at least one embodiment, the interface bus(s) 1110 is not limited to a DMI bus and can include one or more peripheral component interconnection buses (e.g., PCI, PCI Express), memory buses, or other types of interface buses. In at least one embodiment, the processor(s) 1102 include an integrated memory controller 1116 and a platform control hub 1130.In at least one embodiment, the storage controller 1116 enables communication between a storage device and other components of the system 1100, while the platform controller hub (“PCH”) 1130 provides connections to input / output (“I / O”) devices via a local I / O bus.

[0085] In at least one embodiment, the storage device 1120 can be a dynamic random-access memory (DRAM) device, a static random-access memory (SRAM) device, a flash memory device, a phase-change memory device, or some other storage device that has suitable performance to serve as process memory. In at least one embodiment, the storage device 1120 can operate as system memory for the system 1100 to store data 1122 and instructions 1121 that are used when one or more processor(s) 1102 execute an application or process. In at least one embodiment, the memory controller 1116 is also coupled with an optional external graphics processor 1112 that can communicate with one or more graphics processor(s) 1108 in the processors 1102 to perform graphics and media operations.In at least one embodiment, a display device 1111 can be connected to the processor(s) 1102. In at least one embodiment, the display device 1111 can include one or more internal display devices, such as in a mobile electronic device or a laptop device, or external display devices connected via a display interface (e.g., DisplayPort, etc.). In at least one embodiment, the display device 1111 can include a head-mounted display (HMD), such as a stereoscopic display device for use in virtual reality (VR) or augmented reality (AR) applications.

[0086] In at least one embodiment, the platform control hub 1130 enables peripheral devices to connect to the storage device 1120 and the processor 1102 via a high-speed I / O bus. In at least one embodiment, the I / O peripheral devices include, without limitation, an audio controller 1146, a network controller 1134, a firmware interface 1128, a wireless transceiver 1126, touch sensors 1125, and a data storage device 1124 (e.g., a hard disk drive, flash memory, etc.). In at least one embodiment, the data storage device 1124 can be connected via a storage interface (e.g., SATA) or via a peripheral bus, such as a peripheral component interconnection bus (e.g., PCI, PCI Express). In at least one embodiment, the touch sensors 1125 can include touchscreen sensors, pressure sensors, or fingerprint sensors.In at least one embodiment, the wireless transceiver 1126 can be a WiFi transceiver, a Bluetooth transceiver, or a mobile network transceiver, such as a 3G, 4G, or Long-Term Evolution (LTE) transceiver. In at least one embodiment, the firmware interface 1128 enables communication with the system firmware and can, for example, be a Unified Expandable Firmware Interface (UEFI). In at least one embodiment, the network controller 1134 can enable a network connection to a wired network. In at least one embodiment, a high-performance network controller (not shown) is coupled to the interface bus(s) 1110. In at least one embodiment, the audio controller 1146 is a high-resolution, multi-channel audio controller.In at least one embodiment, the system 1100 includes an optional legacy I / O controller 1140 for coupling conventional devices (e.g., Personal System 2 (PS / 2)) to the system. In at least one embodiment, the platform control hub 1130 can also be connected to one or more universal serial bus (“USB”) controller(s) 1142, connection input devices such as keyboard and mouse 1143 combinations, a camera 1144, or other USB input devices.

[0087] In at least one embodiment, an instance of the memory controller 1116 and the platform control hub 1130 can be integrated into a discrete external graphics processor, such as an external graphics processor 1112. In at least one embodiment, the platform control hub 1130 and / or the memory controller 1116 can be external to the one or more processors 1102. For example, in at least one embodiment, the system 1100 can include an external memory controller 1116 and a platform control hub 1130, which can be configured as a memory control hub and peripheral control hub within a system chipset that communicates with the processor(s) 1102.

[0088] The inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. Details regarding the inference and / or training logic 715 are provided below in conjunction with Fig. 7A and / or 7B are provided. In at least one embodiment, sections of or the entire inference and / or training logic 715 can be integrated into a graphics processing unit 1500. For example, in at least one embodiment, the training and / or inference techniques described herein can utilize one or more of the ALUs implemented in a graphics processing unit. Furthermore, in at least one embodiment, the inference and / or training operations described herein can be performed using logic other than that provided in Fig. The logic illustrated in Figures 7A and / or 7B can be performed. In at least one embodiment, weighting parameters can be stored in a memory and / or registers on or off a chip (shown or not shown), which configure the ALU of a graphics processor to execute one or more machine learning algorithms, neural network architectures, use cases, or training techniques described in this document.

[0089] Such components can be used for a content generation pipeline.

[0090] Fig. Figure 12 is a block diagram of a processor 1200 comprising one or more processor core(s) 1202A-1202N, an integrated memory controller 1214, and an integrated graphics processor 1208 according to at least one embodiment. In at least one embodiment, the processor 1200 may include additional cores, up to and including the additional core(s) 1202N, which are represented by dashed boxes. In at least one embodiment, each of the processor cores 1202A-1202N includes one or more internal cache units 1204A-1204N. In at least one embodiment, each processor core also has access to one or more shared cached units 1206.

[0091] In at least one embodiment, the internal cache unit(s) 1204A-1204N and the shared cache unit(s) 1206 constitute a cache memory hierarchy within the processor 1200. In at least one embodiment, the cache memory unit(s) 1204A-1204N can include at least one level of instruction and data cache within each processor core and one or more levels of shared mid-level cache, such as a Level 2 (L2), Level 3 (L3), Level 4 (L4), or other cache levels, wherein a highest cache level in front of the external memory is classified as LLC. In at least one embodiment, cache coherence logic maintains coherence between the various cache units 1206 and 1204A-1204N.

[0092] In at least one embodiment, the processor 1200 can also include a set of one or more bus control units 1216 and a system agent core 1210. In at least one embodiment, one or more bus control units 1216 manage a set of peripheral buses, such as one or more PCI or PCI Express buses. In at least one embodiment, the system agent core 1210 provides a management function for various processor components. In at least one embodiment, the system agent core 1210 includes one or more integrated memory controllers 1214 to manage access to various external storage devices (not shown).

[0093] In at least one embodiment, one or more of the processor cores 1202A-1202N include support for simultaneous multithreading. In at least one embodiment, the system agent core 1210 includes components for coordinating the processor core(s) 1202A-1202N during multithreaded processing. In at least one embodiment, the system agent core 1210 may additionally include a power control unit (PCU) containing logic and components for regulating one or more power states of the processor core(s) 1202A-1202N and the graphics processor 1208.

[0094] In at least one embodiment, the processor 1200 additionally includes a graphics processor 1208 for performing graphics processing operations. In at least one embodiment, the graphics processor 1208 is coupled with a shared cache unit(s) 1206 and the system agent core 1210, which includes one or more integrated memory controllers 1214. In at least one embodiment, the system agent core 1210 also includes a display controller 1211 for driving the graphics processor output to one or more coupled displays. In at least one embodiment, the display controller 1211 can also be a separate module coupled to the graphics processor 1208 via at least one interconnection, or it can be integrated within the graphics processor 1208.

[0095] In at least one embodiment, a ring-based interconnection unit 1212 is used to couple internal components of the processor 1200. In at least one embodiment, an alternative interconnection unit can be used, such as a point-to-point connection, a switched connection, or other methods. In at least one embodiment, the graphics processor 1208 is coupled to the ring interconnection unit 1212 via an I / O connection 1213.

[0096] In at least one embodiment, the I / O connection 1213 represents at least one of several versions of I / O interconnects, including an in-package I / O interconnect that enables communication between different processor components and an embedded high-performance memory module 1218, such as an eDRAM module. In at least one embodiment, each of the processor cores 1202A-1202N and the graphics processor 1208 uses embedded memory modules 1218 as a shared last-level cache.

[0097] In at least one embodiment, the processor core(s) 1202A-1202N are homogeneous cores executing a common instruction set architecture. In at least one embodiment, the processor core(s) 1202A-1202N are heterogeneous in terms of the instruction set architecture (ISA), wherein one or more processor core(s) 1202A-1202N execute a common instruction set, while one or more other cores of the processor core(s) 1202A-1202N execute a subset of a common instruction set or a different instruction set. In at least one embodiment, the processor core(s) 1202A-1202N are heterogeneous in terms of the microarchitecture, wherein one or more cores with relatively higher power consumption are coupled with one or more performance cores with lower power consumption.In at least one embodiment, the 1200 processor can be implemented on one or more chips or as an integrated SoC circuit.

[0098] The inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. Details regarding the inference and / or training logic 715 are provided below in conjunction with Fig. 7A and / or 7B are provided. In at least one embodiment, sections of or all of the inference and / or training logic 715 can be integrated into the processor 1200. For example, in at least one embodiment, the training and / or inference techniques described in this document can use one or more of the ALUs located in a graphics processor 1208, one or more graphics core(s) 1202A-1202N, or other components in Fig. 12 are realized. Furthermore, in at least one embodiment, the inference and / or training operations described herein can be performed using logic other than that described in Fig. The logic illustrated in Figures 7A and / or 7B can be performed. In at least one embodiment, weighting parameters can be stored in a memory and / or registers on or off a chip (shown or not shown), which configure the ALU of a 1200 graphics processor to execute one or more machine learning algorithms, neural network architectures, use cases, or training techniques described in this document.

[0099] Such components can be used for content generation pipelines. VIRTUALIZED COMPUTING PLATFORM

[0100] Fig. Figure 13 is an exemplary data flow diagram for a process 1300 for generating and deploying an image processing and inference pipeline according to at least one embodiment. In at least one embodiment, the process 1300 can be deployed for use with imaging devices, processing devices, and / or other types of equipment in one or more facilities 1302. The process 1300 can be executed in a training system 1304 and / or in a deployment system 1306. In at least one embodiment, the training system 1304 can be used to perform the training, deployment, and implementation of machine learning models (e.g., neural networks, object recognition algorithms, computer vision algorithms, etc.) for use in a deployment system 1306.In at least one embodiment, the deployment system 1306 can be configured to offload processing and computing resources to a distributed computing environment in order to reduce the infrastructure requirements in the facility 1302. In at least one embodiment, one or more applications in a pipeline can use or access services (e.g., inference, visualization, computing, AI, etc.) of the deployment system 1306 during application execution.

[0101] In at least one embodiment, some of the applications used in advanced processing and inference pipelines may employ machine learning models or other AI to perform one or more processing steps. In at least one embodiment, the machine learning models in the facility(ies) 1302 may be trained using data 1308 (such as imaging data) generated in the facility(ies) 1302 (and stored in one or more picture archiving and communication system (PACS) servers in the facility(ies) 1302), or may be trained using imaging or sequencing data 1308 from one or more other facilities or a combination thereof.In at least one embodiment, the training system 1304 can be used to provide applications, services and / or other resources for generating working, deployable machine learning models for the deployment system 1306.

[0102] In at least one embodiment, the model register 1324 can be secured by an object store that can support versioning and object metadata. In at least one embodiment, the object store can be accessed, for example, via a cloud-storage-compatible application programming interface (API) within a cloud platform. In at least one embodiment, the machine learning models within the model register 1324 can be uploaded, listed, edited, or deleted by developers or partners of a system that interacts with an API. In at least one embodiment, an API can provide access to procedures that allow users with appropriate credentials to associate models with applications such that models can be executed as part of the execution of containerized application instantiations.

[0103] In at least one embodiment, the training system 1304 ( Fig. 13) include a scenario in which the facilities 1302 train their own machine learning model or have an existing machine learning model that needs to be optimized or updated. In at least one embodiment, imaging data 1308 generated by one or more imaging devices, sequencing devices, and / or other types of devices can be received. Once the imaging data 1308 has been received, in at least one embodiment, an AI-assisted annotation 1310 can be used to assist in generating annotations corresponding to the imaging data 1308 to be used as ground-truth data for a machine learning model. In at least one embodiment, an AI-assisted annotation 1310 can be used to annotate one or more machine learning models (e.g.,Convolutional neural networks (CNNs) that can be trained to generate annotations corresponding to certain types of imaging data 1308 (e.g., from certain devices) are included. In at least one embodiment, the AI-assisted annotation 1310 can then be used directly or can be adapted or tuned using an annotation tool to generate ground-truth data. In at least one embodiment, the AI-assisted annotation 1310, the labeled data 1312, or a combination thereof can be used as ground-truth data for training a machine learning model. In at least one embodiment, a trained machine learning model can be designated as the output model(s) 1316 and can be used by the deployment system 1306 as described in this document.

[0104] In at least one embodiment, a training pipeline may include a scenario in which the facilities 1302 require a machine learning model for use in performing one or more processing tasks for one or more applications in the deployment system 1306, but the facility(ies) 1302 may not currently have such a machine learning model (or may not have a model that is optimized, efficient, or effective for such purposes). In at least one embodiment, an existing machine learning model may be selected from a model register 1324. In at least one embodiment, the model register 1324 may contain machine learning models that have been trained to perform a variety of different inference tasks on imaging data.In at least one embodiment, the machine learning models in the model register 1324 may have been trained on imaging data from facilities other than the facilities 1302 (e.g., remote facilities). In at least one embodiment, machine learning models may have been trained on image data from one site, two sites, or any number of sites. In at least one embodiment, when trained on imaging data from a specific site, the training may take place at that site or at least in a manner that protects the confidentiality of the imaging data or restricts the transmission of the imaging data out of the facility. Once a model has been trained—or partially trained—at a site, a machine learning model may, in at least one embodiment, be added to the model register 1324.In at least one embodiment, a machine learning model can then be retrained or updated on any number of other facilities, and a retrained or updated model can be made available in the model register 1324. In at least one embodiment, a machine learning model can then be selected from the model register 1324—and designated as output model(s) 1316—and can be used in the deployment system 1306 to perform one or more processing tasks for one or more applications of a deployment system.

[0105] In at least one embodiment, a scenario may include facilities 1302 that require a machine learning model for use in performing one or more processing tasks for one or more applications in the deployment system 1306, but the facilities 1302 may not currently have such a machine learning model (or may not have a model that is optimized, efficient, or effective for such purposes). In at least one embodiment, a machine learning model selected from the model register 1324 may not be matched or optimized for imaging data 1308 generated in the facilities 1302 due to differences in the populations, the robustness of the training data used to train a machine learning model, the diversity of anomalies in the training data, and / or other problems with the training data.In at least one embodiment, an AI-assisted annotation 1310 can be used to help generate annotations corresponding to the imaging data 1308, which are to be used as ground-truth data for retraining or updating a machine learning model. In at least one embodiment, the labeled data 1312 can be used as ground-truth data for training a machine learning model. In at least one embodiment, the retraining or updating of a machine learning model can be referred to as model training 1314. In at least one embodiment, the model training 1314—e.g., AI-assisted annotation 1310, labeled data 1312, or a combination thereof—can be used as ground-truth data for retraining or updating a machine learning model.In at least one embodiment, a trained machine learning model can be designated as output model(s) 1316 and can be used by the deployment system 1306 as described in this document.

[0106] In at least one embodiment, the deployment system 1306 can include software 1318, services 1320, hardware 1322, and / or other components, features, and functionality. In at least one embodiment, the deployment system 1306 can include a software "stack" such that the software 1318 can be built on top of services 1320 and can use the services 1320 to perform some or all of the processing tasks and services 1320, and the software 1318 can be built on top of hardware 1322 and can use the hardware 1322 to perform processing, storage, and / or other computing tasks of the deployment system 1306. In at least one embodiment, the software 1318 can include any number of different containers, each container being capable of executing an instantiation of an application.In at least one embodiment, each application can perform one or more processing tasks in an advanced processing and inference pipeline (e.g., inference, object detection, feature recognition, segmentation, image enhancement, calibration, etc.). In at least one embodiment, an advanced processing and inference pipeline can be defined based on selections of various containers that are desired or required for processing imaging data 1308, in addition to containers that receive and configure imaging data for use by each container and / or for use by the facility 1302 after processing via a pipeline (e.g., to convert outputs back into a usable data type).In at least one embodiment, a combination of containers within the software 1318 (which, for example, forms a pipeline) can be referred to as a virtual instrument (as will be described in more detail in this document), and a virtual instrument can utilize services 1320 and hardware 1322 to perform one or all of the processing tasks of applications instantiated in containers.

[0107] In at least one embodiment, a data processing pipeline can receive input data (e.g., imaging data 1308) in a specific format in response to an inference request (e.g., a request from a user of the deployment system 1306). In at least one embodiment, the input data can be representative of one or more images, video footage, and / or other data representations generated by one or more imaging devices. In at least one embodiment, the data can be preprocessed as part of the data processing pipeline to prepare the data for processing by one or more applications.In at least one embodiment, post-processing can be performed on the output of one or more inference tasks or other processing tasks of a pipeline to prepare output data for a subsequent application and / or to prepare output data for transmission and / or use by a user (e.g., as a response to an inference request). In at least one embodiment, the inference tasks can be performed by one or more machine learning models, such as trained or provisioned neural networks, which may include output model(s) 1316 of the training system 1304.

[0108] In at least one embodiment, the tasks of the data processing pipeline can be encapsulated in (one) container(s), each representing a discrete, fully functional instantiation of an application and a virtualized computing environment capable of referencing machine learning models. In at least one embodiment, the containers or applications can be published to a private area (e.g., with restricted access) of a container register (described in more detail below in this document), and trained or deployed models can be stored in the model register 1324 and assigned to one or more applications. In at least one embodiment, images of applications (e.g.,Container images) are available in a container registry, and once they have been selected by a user from a container registry for use in a pipeline, an image can be used to create a container for instantiating an application for use by a user's system.

[0109] In at least one embodiment, developers (e.g., software developers, clinicians, physicians, etc.) can develop, publish, and store applications (e.g., in the form of containers) for performing image processing and / or inference on provided data. In at least one embodiment, the development, publication, and / or storage can be performed using a software development kit (SDK) associated with a system (e.g., to ensure that a developed application and / or container is compliant or compatible with a system). In at least one embodiment, a developed application can be tested locally (e.g., in a first facility with data from a first facility) using an SDK that includes at least some of the services 1320 as a system (e.g., the system 1200 from Fig. 12) can support. In at least one embodiment, since DICOM objects can contain any number between one and hundreds of images or other data types, and due to variations in the data, a developer may be responsible for managing (e.g., setting constructs for, incorporating preprocessing into an application, etc.), extracting, and preparing incoming data. In at least one embodiment, once validated by process 1300 (e.g., for accuracy), an application may be available in a container register for selection and / or implementation by a user to perform one or more processing tasks with respect to data in a user's facility (e.g., a second facility).

[0110] In at least one embodiment, developers can then make applications or containers accessible and usable by users of a system (e.g., the System 1300 from) via a network. Fig. 13) share. In at least one embodiment, completed and validated applications or containers can be stored in a container register, and associated machine learning models can be stored in the model register 1324. In at least one embodiment, a requesting entity—providing an inference or image processing request—can search a container register and / or a model register 1324 for an application, container, dataset, machine learning model, etc., select a desired combination of elements to include in the data processing pipeline, and submit an image processing request.In at least one embodiment, a request may include input data (and in some examples, associated patient data) required to execute a request, and / or a selection of machine learning applications and / or models to be executed when processing a request. In at least one embodiment, a request may then be passed to one or more components of the deployment system 1306 (e.g., a cloud) to perform processing in the data processing pipeline. In at least one embodiment, the processing by the deployment system 1306 may involve referencing selected elements (e.g., applications, containers, models, etc.) from a container register and / or model register 1324. In at least one embodiment, once results are generated by a pipeline, results may be provided to a user as a reference (e.g.,(for viewing in a viewing application suite running on a local workstation or terminal).

[0111] In at least one embodiment, services 1320 can be used to assist in the processing or execution of applications or containers in pipelines. In at least one embodiment, the services 1320 can include computing services, artificial intelligence (AI) services, visualization services, and / or other types of services. In at least one embodiment, the services 1320 can provide functionality that is common to one or more applications in the software 1318, such that the functionality can be abstracted to a service that can be called or used by the applications. In at least one embodiment, the functionality provided by the services 1320 can be executed dynamically and more efficiently, and is also highly scalable by allowing the applications to process data in parallel (e.g., using a parallel computing platform 1230). Fig. 12)) to process. In at least one embodiment, instead of each application sharing the same functionality offered by Services 1320 having its own instance of Services 1320, Services 1320 can be shared by various applications. In at least one embodiment, the Services can include an inference server or inference engine that can be used to perform detection or segmentation tasks, as non-limiting examples. In at least one embodiment, a model training service can be included that can provide the ability to train and / or retrain machine learning models. In at least one embodiment, a data augmentation service can also be included that enables the extraction, resizing, scaling, and / or other enhancement of GPU-accelerated data (e.g.,DICOM data, RIS data, CIS data, REST-compliant data, RPC data, raw data, etc.) can be provided. In at least one embodiment, a visualization service can be used that can add image rendering effects—such as ray tracing, rasterization, denoising, sharpening, etc.—to make two-dimensional (2D) and / or three-dimensional (3D) models more realistic. In at least one embodiment, services for virtual instruments can be included that provide beam shaping, segmentation, inference, imaging, and / or support for other applications within virtual instrument pipelines.

[0112] In at least one embodiment, where Services 1320 include an AI service (e.g., an inference service), one or more machine learning models can be executed by invoking an inference service (e.g., an inference server) (e.g., as an API call) to execute one or more machine learning models or their processing as part of application execution. In at least one embodiment, where another application includes one or more machine learning models for segmentation tasks, an application can invoke an inference service to execute machine learning models to perform one or more of the operations associated with segmentation tasks.In at least one embodiment, the software 1318, which implements the advanced processing and inference pipeline that includes a segmentation application and anomaly detection application, can be streamlined because each application can call the same inference service to perform one or more inference tasks.

[0113] In at least one embodiment, the hardware 1322 can include a GPU, CPU, graphics cards, an AI / deep learning system (e.g., an AI supercomputer such as NVIDIA's DGX), a cloud platform, or a combination thereof. In at least one embodiment, different types of hardware 1322 can be used to provide efficient, custom-built software 1318 and services 1320 in the deployment system 1306. In at least one embodiment, the use of GPU processing for local processing (e.g., in the facility 1302) within an AI / deep learning system, in a cloud system, and / or in other processing components of the deployment system 1306 can be implemented to improve the efficiency, accuracy, and effectiveness of image processing and generation.In at least one embodiment, the software 1318 and / or services 1320 can be optimized for GPU processing with respect to deep learning, machine learning, and / or high-performance computing, as non-limiting examples. In at least one embodiment, at least part of the computing environment of the deployment system 1306 and / or the training system 1304 can be run in a data center, on one or more supercomputers, or high-performance computing systems with GPU-optimized software (e.g., a hardware and software combination of NVIDIA's DGX system). In at least one embodiment, the hardware 1322 can include any number of GPUs that can be called upon to perform parallel data processing as described in this document.In at least one embodiment, the cloud platform may further include GPU processing for GPU-optimized execution of deep learning tasks, machine learning tasks, or other computational tasks. In at least one embodiment, the cloud platform (e.g., NVIDIA's NGC) may be implemented using one or more AI / deep learning supercomputers and / or GPU-optimized software (e.g., NVIDIA's DGX systems) as a hardware abstraction and scaling platform. In at least one embodiment, the cloud platform may integrate an application container clustering or orchestration system (e.g., Kubernetes) across multiple GPUs to enable seamless scaling and load balancing.

[0114] Fig. Figure 14 is a system diagram for an exemplary system 1400 for generating and providing an imaging deployment pipeline according to at least one embodiment. In at least one embodiment, the system 1400 can be used to perform the process 1300 from Fig. 13 and / or other processes that include advanced processing and inference pipelines. In at least one embodiment, the system 1400 can include the training system 1304 and the deployment system 1306. In at least one embodiment, the training system 1304 and the deployment system 1306 can be implemented using software 1318, services 1320, and / or hardware 1322, as described in this document.

[0115] In at least one embodiment, System 1400 (e.g., Training System 1304 and / or Deployment System 1306) can be implemented in a cloud computing environment (e.g., using Cloud 1426). In at least one embodiment, System 1400 can be implemented locally with respect to a healthcare facility or as a combination of both cloud and local computing resources. In at least one embodiment, access to APIs in Cloud 1426 can be restricted to authorized users by mandated security measures or protocols. In at least one embodiment, a security protocol can include web tokens that can be signed by an authentication service (e.g., AuthN, AuthZ, Gluecon, etc.) and carry appropriate authorization.In at least one embodiment, the APIs of virtual instruments (described in this document) or other instantiations of System 1400 can be restricted to a set of public IPs that have been verified or authorized for interaction.

[0116] In at least one embodiment, various components of the System 1400 can communicate with each other using one of several different network types, including, without limitation, local area networks (LANs) and / or wide area networks (WANs), via wired and / or wireless communication protocols. In at least one embodiment, communication between devices and components of the System 1400 (e.g., for transmitting inference requests, receiving results of inference requests, etc.) can be carried out via one or more data buses, wireless data protocols (WiFi), wired data protocols (e.g., Ethernet), etc.

[0117] In at least one embodiment, the training system 1304 can execute training pipelines 1404, similar to those described in this document with respect to Fig. 13 are described. In at least one embodiment, in which one or more machine learning models are to be used in the deployment pipeline(s) 1410 by the deployment system 1306, the training pipelines 1404 can be used to train or retrain one or more (e.g., pre-trained) models, and / or to implement one or more of the pre-trained models 1406 (e.g., without requiring retraining or an update). In at least one embodiment, output model(s) 1316 can be generated as a result of the training pipelines 1404. In at least one embodiment, the training pipelines 1404 can include any number of processing steps, such as, without limitation, the conversion or adaptation of imaging data (or other input data).In at least one embodiment, different training pipelines 1404 can be used for different machine learning models employed by the deployment system 1306. In at least one embodiment, a training pipeline (or pipelines) 1404 similar to a first example, which is described with respect to . Fig. As described in section 13, a training pipeline (or pipelines) 1404, similar to a second example, can be used for a first machine learning model, which is similar to a second example that is similar to Fig. 13, can be used for a second machine learning model, and training pipelines 1404, which are similar to a third example, can be used in relation to Fig. The training system 1304, as described in section 13, can be used for a third machine learning model. In at least one embodiment, any combination of tasks within the training system 1304 can be used, depending on the requirements of each machine learning model. In at least one embodiment, one or more of the machine learning models can already be trained and ready for deployment, so that the machine learning models may not require processing by the training system 1304 and can be implemented by the deployment system 1306.

[0118] In at least one embodiment, output models 1316 and / or pretrained model(s) 1406 may include any type of machine learning model(s) depending on the implementation or embodiment. In at least one embodiment and without limitation, the machine learning models used by the System 1400 may include one or more machine learning models that employ linear regression, logistic regression, decision trees, support vector machines (SVMs), Naive Bayes, k-nearest neighbors (Knn), K-means clustering, random forest, dimensionality reduction algorithms, gradient enhancement algorithms, neural networks (e.g., self-encoder, convolution, recursive, perceptron, long-term / short-term memory (LSTM), Hopfield, Boltzmann, deep belief, unfolding, generative contradictory, fluid state machine, etc.) and / or other types of machine learning models.

[0119] In at least one embodiment, the training pipelines 1404 can include AI-assisted annotation, as described in this document at least with respect to Fig. 14 described in more detail. In at least one embodiment, the labeled data 1312 (e.g., conventional annotation) can be generated by any number of techniques. In at least one embodiment, labels or other annotations can be generated in a drawing program (e.g., an annotation program), a CAD (Computer Aided Design) program, a marking program, or another type of program suitable for generating annotations or labels for ground truth, and / or they can be drawn by hand in some examples. In at least one embodiment, ground truth data can be generated synthetically (e.g., generated from computer models or renderings), generated in reality (e.g., designed and generated from real data), or generated automatically by machine (e.g.,using feature analysis and learning to extract features from data and then generate labels), be annotated by humans (e.g., labeler or annotation expert, defines the position of labels), and / or a combination thereof. In at least one embodiment, for each instance of imaging data 1308 (or other data type used by machine learning models), there can be corresponding ground-truth data generated by the training system 1304. In at least one embodiment, AI-assisted annotation can be performed as part of the deployment pipelines 1410, either in addition to or instead of AI-assisted annotation included in the training pipelines 1404. In at least one embodiment, the system 1400 can include a multi-layered platform that includes a software layer (e.g.,The software 1318) of diagnostic applications (or other types of applications) that can perform one or more medical imaging and diagnostic functions. In at least one embodiment, the system 1400 can be communicatively coupled to PACS server networks of one or more facilities (e.g., via encrypted links). In at least one embodiment, the system 1400 can be configured to access and reference data from PACS servers in order to perform operations such as training machine learning models, deploying machine learning models, image processing, inference, and / or other operations.

[0120] In at least one embodiment, a software layer can be implemented as a secure, encrypted, and / or authenticated API through which applications or containers can be accessed (e.g., called) by one or more external environments (e.g., facility 1302). In at least one embodiment, the applications can then call or execute one or more services 1320 to perform computational, AI, or visualization tasks assigned to the respective applications, and the software 1318 and / or services 1320 can utilize hardware 1322 to perform processing tasks effectively and efficiently. In at least one embodiment, communications sent to or received from a training system 1304 and a deployment system 1306 can be conducted using a pair of DICOM adapters 1402A, 1402B.

[0121] In at least one embodiment, the deployment system 1306 can execute deployment pipeline(s) 1410. In at least one embodiment, the deployment pipeline(s) 1410 can include any number of applications that can be applied sequentially, non-sequentially, or otherwise to imaging data (and / or other types of data) generated by imaging devices, sequencing devices, genomics devices, etc. – including AI-assisted annotation, as described above. In at least one embodiment, as described in this document, deployment pipeline(s) 1410 for a single device can be designated as a virtual instrument for a device (e.g., a virtual ultrasound instrument, a virtual CT scan instrument, a virtual sequencing instrument, etc.).In at least one embodiment, there can be more than one deployment pipeline 1410 for a single device, depending on the information desired from the data generated by the device. In at least one embodiment where anomaly detection from an MRI machine is desired, there can be a first deployment pipeline 1410, and where image enhancement from the output of an MRI machine is desired, there can be a second deployment pipeline 1410.

[0122] In at least one embodiment, an image generation application may include a processing task that involves the use of a machine learning model. In at least one embodiment, a user may choose to use their own machine learning model or select a machine learning model from model register 1324. In at least one embodiment, a user may implement their own machine learning model or select a machine learning model for inclusion in an application to perform a processing task. In at least one embodiment, applications may be selectable and customizable, and by defining application constructs, the provisioning and implementation of applications for a particular user are presented as a more seamless user experience.In at least one embodiment, by utilizing other features of the system 1400 - such as the services 1320 and the hardware 1322 - the deployment pipeline(s) 1410 can be even more user-friendly, provide simpler integration and deliver more accurate, efficient and timely results.

[0123] In at least one embodiment, the deployment system 1306 may include a user interface (“UI”)1414 (e.g., a graphical user interface, a web interface, etc.) that can be used to select applications for inclusion in one or more deployment pipeline(s) 1410, to arrange applications, to edit or modify applications or parameters or constructs thereof, to use and interact with one or more deployment pipeline(s) 1410 during setup and / or deployment, and / or to otherwise interact with the deployment system 1306.In at least one embodiment, which is not illustrated with respect to the training system 1304, the user interface 1414 (or another user interface) can be used to select models for use with the deployment system 1306, to select models for training or retraining in the training system 1304, and / or to otherwise interact with the training system 1304.

[0124] In at least one embodiment, the pipeline manager 1412 can be used in addition to an application orchestration system 1428 to manage the interaction between applications or containers of the deployment pipeline(s) 1410 and services 1320 and / or hardware 1322. In at least one embodiment, the pipeline manager 1412 can be configured to allow application-to-application, application-to-services 1320, and / or application-or-service-to-hardware 1322 interactions. In at least one embodiment, although illustrated as being included in the software 1318, this is not intended to be restrictive, and in some examples, the pipeline manager 1412 can be included in the services 1320. In at least one embodiment, the application orchestration system 1428 (e.g., Kubernetes, Docker, etc.) can be configured to allow application-to-application, application-to-services 1320, and / or application-to-services 1322 interactions.) include a container coordination system that can group applications into containers as logical units for coordination, management, scaling, and deployment. In at least one embodiment, by assigning applications from the deployment pipeline(s) 1410 (e.g., a reconstruction application, a segmentation application, etc.) to individual containers, each application can be run in a self-contained environment (e.g., at a kernel level) to increase speed and efficiency.

[0125] In at least one embodiment, each application and / or container (or an image thereof) can be developed, modified, and deployed individually (e.g., a first user or developer can develop, modify, and deploy a first application, and a second user or developer can develop, modify, and deploy a second application separately from the first user or developer), making it possible to focus on and attend to the task of a single application and / or one or more containers without being hindered by tasks of one or more other applications or containers. In at least one embodiment, communication and interaction between different containers or applications can be supported by the pipeline manager 1412 and the application orchestration system 1428.In at least one embodiment, as long as an expected input and / or output of each container or application is known to a system (e.g., based on constructs of applications or containers), the application orchestration system 1428 and / or the pipeline manager 1412 can enable communication and resource sharing between each of the applications or containers. In at least one embodiment, since one or more of the applications or containers in the deployment pipeline(s) 1410 can share the same services and resources, the application orchestration system 1428 can coordinate services or resources between various applications or containers, balance their load, and determine their shared use.In at least one embodiment, a scheduler can be used to track the resource requirements of applications or containers, the current or planned use of these resources, and resource availability. In at least one embodiment, a scheduler can thus allocate resources to different applications and distribute them between and among the applications, taking into account the requirements and availability of a system. In some examples, a scheduler (and / or another component of the application orchestration system 1428) can determine the availability and distribution of resources based on constraints imposed on a system (e.g., user constraints), such as quality of service (QoS), urgency of the need for data output (e.g., to determine whether real-time or delayed processing is required), and so on.

[0126] In at least one embodiment, service(s) 1320, which are used and shared by applications or containers in the deployment system 1306, can include compute service(s) 1416, AI service(s) 1418, visualization service(s) 1420, and / or other service types. In at least one embodiment, applications can call (e.g., execute) one or more of the services 1320 to perform processing operations for an application. In at least one embodiment, compute services 1416 can be used by applications to perform superdata processing or other high-performance computing (HPC) tasks. In at least one embodiment, one or more compute services 1416 can be used to perform parallel processing (e.g.,using a parallel computing platform 1430) for processing data across one or more applications and / or performing one or more tasks of a single application substantially simultaneously. In at least one embodiment, the parallel computing platform 1430 (e.g., NVIDIA's CUDA) can enable general-purpose computing on GPUs (GPGPU) (e.g., GPU / graphics cards 1422). In at least one embodiment, a software layer of a parallel computing platform 1430 can provide access to virtual instruction sets and parallel computing elements of GPUs for executing computing kernels. In at least one embodiment, the parallel computing platform 1430 can include memory, and in some embodiments, memory can be shared by multiple containers and / or among different processing tasks within a single container.In at least one embodiment, inter-process communication (IPC) calls can be generated for multiple containers and / or for multiple processes within a container to use the same data from a shared memory segment of the Parallel Computing Platform 1430 (e.g., where several different stages of an application or multiple applications process the same information). In at least one embodiment, instead of creating a copy of data and moving data to different memory locations (e.g., a read / write operation), the same data can be used in the same memory location for any number of processing tasks (e.g., concurrently, at different times, etc.).In at least one embodiment, since data is used to generate new data as a result of processing, this information can be stored in a new data location and shared by different applications. In at least one embodiment, the location of data and the location of updated or modified data can be part of a definition of how a payload is understood within containers.

[0127] In at least one embodiment, AI services 1418 can be used to perform inference services for executing one or more machine learning models assigned to applications (e.g., tasked with executing one or more processing tasks of an application). In at least one embodiment, the AI ​​service(s) 1418 can support an AI system 1424 to execute one or more machine learning models (e.g., neural networks, such as CNNs) for segmentation, reconstruction, object detection, feature recognition, classification, and / or other inference tasks. In at least one embodiment, the applications of the deployment pipeline(s) 1410 can use one or more output models 1316 from the training system 1304 and / or other application models to perform inference on imaging data.In at least one embodiment, two or more examples of inference using the Application Orchestration System 1428 (e.g., a scheduler) may be available. In at least one embodiment, a first category may include a high-priority / low-latency path that can achieve higher Service Level Agreements, such as for performing inferences on urgent requests during an emergency or for a radiologist during diagnosis. In at least one embodiment, a second category may include a standard-priority path that can be used for requests that may not be urgent or where analysis can be performed at a later time. In at least one embodiment, the Application Orchestration System 1428 may allocate resources (e.g.,Distribute services 1320 and / or hardware 1322) based on priority paths for different inference tasks of AI service(s) 1418.

[0128] In at least one embodiment, a shared memory for AI service(s) 1418 can be installed in the system 1400. In at least one embodiment, the shared memory can function as a cache (or other type of device) and be used to process requests from applications. In at least one embodiment, when an inference request is made, a request can be received by a set of API instances of the deployment system 1306, and one or more instances can be selected (e.g., for best fit, load balancing, etc.) to process a request.In at least one embodiment, to process a request, a request can be entered into a database; a machine learning model can be located from the model register 1324 if it is not already in the cache; a validation step can ensure that the appropriate machine learning model is loaded into a cache (e.g., shared memory); and / or a copy of a model can be stored in a cache. In at least one embodiment, a scheduler (e.g., of the pipeline manager 1412) can be used to start an application referenced in a request if an application is not already running or if there are not enough instances of an application. In at least one embodiment, an inference server can be started if it has not already been started to execute a model.Any number of inference servers can be started per model. In at least one embodiment, models can be cached in a pull model where inference servers form a cluster, if load balancing is advantageous. In at least one embodiment, the inference servers can be statically loaded onto corresponding distributed servers.

[0129] In at least one embodiment, inference can be performed using an inference server running in a container. In at least one embodiment, an instance of an inference server can be associated with a model (and optionally a plurality of versions of a model). In at least one embodiment, if an instance of an inference server does not exist when a request to perform inference on a model is received, a new instance can be loaded. In at least one embodiment, when an inference server is started, a model can be passed to an inference server, so that the same container can be used to handle different models, as long as the inference server is running as a separate instance.

[0130] In at least one embodiment, an inference request for a given application can be received during application execution, and a container (which, for example, hosts an instance of an inference server) can be loaded (if not already done) and a start procedure invoked. In at least one embodiment, the preprocessing logic in a container can load incoming data, decode it, and / or perform any additional preprocessing on it (e.g., using CPU(s) and / or GPU(s)). In at least one embodiment, once the data is prepared for inference, a container can perform the inference on the data as needed. In at least one embodiment, this can involve a single inference call on a single image (e.g., a hand X-ray) or require inference on hundreds of images (e.g., a breast CT scan).In at least one embodiment, an application can summarize the results before completion, which can include, without limitation, a single confidence score, pixel-level segmentation, voxel-level segmentation, the generation of a visualization, or the generation of text summarizing findings. In at least one embodiment, different models or applications can be assigned different priorities. For example, some models can have a real-time priority (TAT < 1 min), while others can have a lower priority (e.g., TAT < 10 min). In at least one embodiment, model execution times can be measured by the requesting institution or organization and can include partner network traversal time as well as execution by an inference service.

[0131] In at least one embodiment, the transmission of requests between Services 1320 and inference applications can be hidden behind a software development kit (SDK), and robust transport can be provided via a queue. In at least one embodiment, a request via an API is placed in a queue for a unique application / tenant ID combination, and an SDK pulls a request from the queue and sends it to an application. In at least one embodiment, a queue name can be provided in an environment from which an SDK retrieves it. In at least one embodiment, asynchronous communication via a queue can be beneficial because it allows each instance of an application to begin work as soon as it becomes available. Results can be pushed back via a queue to ensure that no data is lost.In at least one embodiment, queues can also provide the ability to segment work, since the highest-priority work can go to a queue with the most instances of an associated application, while the lowest-priority work can go to a queue with a single associated instance that processes tasks in a received order. In at least one embodiment, an application can run on a GPU-accelerated instance generated in Cloud 1426, and an inference service can perform the inference on a GPU.

[0132] In at least one embodiment, a visualization service (or services) 1420 can be used to generate visualizations for visualizing application output and / or a deployment pipeline (or pipelines) 1410. In at least one embodiment, the GPUs / graphics cards 1422 can be used by a visualization service (or services) 1420 to generate visualizations. In at least one embodiment, rendering effects, such as ray tracing, can be implemented by a visualization service (or services) 1420 to generate higher-quality visualizations. In at least one embodiment, visualizations can include, without limitation, 2D image rendering, 3D volume rendering, 3D volume reconstruction, 2D tomographic sections, virtual reality displays, augmented reality displays, etc. In at least one embodiment, virtualized environments can be used to provide a virtually interactive display or environment (e.g.,to generate a virtual environment for interaction by users of a system (e.g., doctors, nurses, radiologists, etc.). In at least one embodiment, visualization service(s) 1420 may include an internal visualizer, film technology, and / or other rendering or image processing capabilities or functionality (e.g., ray tracing, rasterization, internal optics, etc.).

[0133] In at least one embodiment, the hardware 1322 may include GPUs / graphics cards 1422, an AI system 1424, a cloud 1426, and / or any other hardware used to run a training system 1304 and / or a deployment system 1306. In at least one embodiment, the GPUs / graphics cards 1422 (e.g., NVIDIA's TESLA and / or QUADRO GPUs) may include any number of GPUs that can be used to perform processing tasks of compute service(s) 1416, AI service(s) 1418, visualization service(s) 1420, other services, and / or any of the features or functionality of the software 1318.For example, with respect to AI service(s) 1418, the GPUs / graphics cards 1422 can be used to perform preprocessing on imaging data (or other types of data used by machine learning models), postprocessing on outputs from machine learning models, and / or to perform inference (e.g., to run machine learning models). In at least one embodiment, the cloud 1426, the AI ​​system 1424, and / or other components of the system 1400 can use the GPUs / graphics cards 1422. In at least one embodiment, the cloud 1426 can include a GPU-optimized platform for deep learning tasks. In at least one embodiment, the AI ​​system 1424 can use GPUs, and the cloud 1426—or at least a part of it that is tasked with deep learning or inference—can run using one or more AI systems 1424.Although the Hardware 1322 is depicted as discrete components, this is not intended as a limitation, and any components of Hardware 1322 can be combined or used with any other components of Hardware 1322.

[0134] In at least one embodiment, the AI ​​system 1424 may include a specially manufactured computer system (e.g., a supercomputer or an HPC) configured for inference, deep learning, machine learning, and / or other artificial intelligence tasks. In at least one embodiment, the AI ​​system 1424 (e.g., NVIDIA's DGX) may include GPU-optimized software (e.g., a software stack) that can be run using a variety of GPUs / graphics cards 1422 in addition to the CPU, RAM, memory, and / or other components, features, or functionality. In at least one embodiment, one or more AI systems 1424 may be deployed in a cloud 1426 (e.g., in a data center) to perform one or all of the AI-based processing tasks of the system 1400.

[0135] In at least one embodiment, Cloud 1426 can include a GPU-accelerated infrastructure (e.g., NVIDIA NGC) that can provide a GPU-optimized platform for performing processing tasks of System 1400. In at least one embodiment, Cloud 1426 can include an AI system 1424 for performing one or more of the AI-based tasks of System 1400 (e.g., as a hardware abstraction and scaling platform). In at least one embodiment, Cloud 1426 can be integrated with the application orchestration system 1428 and utilize multiple GPUs to enable seamless scaling and load balancing among the applications and services 1320. In at least one embodiment, Cloud 1426 can be tasked with performing at least some of the services 1320 of System 1400, including the compute service(s) 1416, the AI ​​service(s) 1418, and / or the...The visualization service(s) 1420 are included, as described in this document. In at least one embodiment, the cloud 1426 can perform small and large batch inference (running, for example, NVIDIA's TENSOR RT), provide an accelerated parallel computing API and platform 1430 (e.g., NVIDIA's CUDA), run the application orchestration system 1428 (e.g., Kubernetes), provide a graphics rendering API and platform (e.g., for ray tracing, 2D graphics, 3D graphics, and / or other rendering techniques to produce higher-quality movies), and / or provide other functionality for the system 1400.

[0136] Fig. Figure 15A illustrates a data flow diagram for a process 1500 for training, retraining, or updating a machine learning model according to at least one embodiment. In at least one embodiment, the process 1500 can be defined as a non-limiting example using the system 1400 from Fig. 14. In at least one embodiment, the process 1500 can utilize the services and / or hardware as described in this document. In at least one embodiment, refined models 1512, generated by the process 1500, can be executed by a deployment system for one or more containerized applications in the deployment pipelines.

[0137] In at least one embodiment, the model training 1514 can involve retraining or updating an initial model 1504 (e.g., a pre-trained model) using new training data (e.g., new input data, such as a customer record 1506, and / or new ground-truth data associated with the input data). In at least one embodiment, retraining or updating the initial model 1504 can involve resetting, deleting, and / or replacing output or loss layer(s) of the initial model 1504 with updated or new output or loss layer(s). In at least one embodiment, the initial model 1504 can have previously fine-tuned parameters (e.g.,Weightings and / or systematic errors) left over from previous training, so that training or retraining 1514 does not take as long or require as much processing power as training a model from scratch. In at least one embodiment, during model training, by resetting or replacing the output or loss layer(s) of the initial model 1504, the parameters for a new dataset can be updated and recalibrated based on loss calculations associated with the accuracy of the output or loss layer(s) when generating predictions on a new customer dataset 1506.

[0138] In at least one embodiment, pre-trained models 1506 may be stored in a data archive or register. In at least one embodiment, the pre-trained models 1506 may have been trained, at least partially, in one or more facilities other than the facility performing the process 1500. In at least one embodiment, to protect the privacy and rights of patients, trial participants, or customers from different facilities, the pre-trained models 1506 may have been trained on-site using customer or patient data generated on-site.In at least one embodiment, the pre-trained models 1506 can be trained using a cloud and / or other hardware, but confidential, privacy-protected patient data must not be transmitted to, used by, or accessible to any component of a cloud (or other hardware outside the user's own premises). In at least one embodiment, in which the pre-trained models 1506 are trained using patient data from more than one institution, the pre-trained models 1506 may have been trained individually for each institution before training on patient or customer data from another institution. In at least one embodiment, such as when customer or patient data is used for privacy reasons (e.g., by waiver, for experimental purposes, etc.),) have been released, or if customer or patient data is included in a public dataset, customer or patient data can be used by any number of facilities to train the pre-trained 1506 models on-site and / or off-site, such as in a data center or other cloud computing infrastructure.

[0139] In at least one embodiment, when selecting applications for use in the deployment pipelines, a user can also select machine learning models to be used for specific applications. In at least one embodiment, a user may not have a model available for use, allowing the user to select the pre-trained model for use with an application. In at least one embodiment, the pre-trained model may not be optimized to generate accurate results on a user's facility's customer dataset 1506 (e.g., based on patient diversity, demographic data, types of medical imaging devices used, etc.).In at least one embodiment, the pre-trained model can be updated, retrained and / or fine-tuned before being deployed in a deployment pipeline for use with one or more applications in a given facility.

[0140] In at least one embodiment, a user can select the pre-trained model(s) to be updated, retrained, and / or fine-tuned, and this pre-trained model can be referred to as the initial model 1504 for a training system within the process 1500. In at least one embodiment, a customer dataset 1506 (e.g., imaging data, genomic data, sequencing data, or other types of data generated by devices in a facility) can be used to perform model training (which may, without limitation, include transfer learning) on ​​the initial model 1504 to generate a refined model 1512. In at least one embodiment, ground-truth data corresponding to the customer dataset 1506 can be generated by the training system 1304.In at least one embodiment, ground truth data can be generated at least partially by clinicians, scientists, physicians, practitioners, in an institution.

[0141] In at least one embodiment, AI-assisted annotation can be used in some examples to generate ground-truth data. In at least one embodiment, the AI-assisted annotation 1310 (e.g., implemented using an AI-assisted annotation SDK) can utilize machine learning models (e.g., neural networks) to generate suggested or predicted ground-truth data for a customer dataset. In at least one embodiment, the user can use annotation tools within a user interface (a graphical user interface (GUI)) on the computing device.

[0142] In at least one embodiment, the user 1510 can interact with a GUI via the computing device 1508 to edit or fine-tune (automatic) annotations. In at least one embodiment, a polygon editing feature can be used to move the vertices of a polygon to more accurate or fine-tuned positions.

[0143] In at least one embodiment, the ground truth data (e.g., from AI-assisted annotation, manual labeling, etc.) can be used during model training to generate the refined model 1512 once the customer dataset 1506 has associated ground truth data. In at least one embodiment, the customer dataset 1506 can be applied to an initial model 1504 any number of times, and the ground truth data can be used to update the parameters of the initial model 1504 until an acceptable level of accuracy is achieved for the refined model 1512. In at least one embodiment, once the refined model 1512 has been generated, it can be deployed within one or more deployment pipelines in a facility to perform one or more processing tasks related to medical imaging data.

[0144] In at least one embodiment, the refined model 1512 can be uploaded to pretrained models in a model register for selection by another facility. In at least one embodiment, this process can be completed with any number of facilities, allowing the refined model 1512 to be further refined on new datasets any number of times to generate a more universal model.

[0145] Fig. Figure 15B is an exemplary illustration of a client-server architecture 1532 for enhancing annotation tools with pre-trained annotation models, according to at least one embodiment. In at least one embodiment, an AI-supported annotation tool 1536 can be instantiated based on a client-server architecture 1532. In at least one embodiment, the annotation tool 1536 can, for example, assist radiologists in imaging applications to identify organs and anomalies. In at least one embodiment, the imaging applications can include software tools that assist the user 1510, as a non-restrictive example, in identifying a few extreme points on a specific organ of interest in raw images 1534 (e.g., in a 3D MRI or CT scan) and obtain automatically labeled results for all 2D slices of a specific organ.In at least one embodiment, the results can be stored in a data archive as training data 1538 and (by way of example and without limitation) used as ground-truth data for training. In at least one embodiment, if the computing device 1508 sends extreme points for AI-assisted annotation, a deep learning model can, for example, receive this data as input and return inference results of a segmented organ or anomaly. In at least one embodiment, pre-instantiated annotation tools, such as the AI-assisted annotation tool 1536, can be used. Fig.15B, can be improved by making API calls (e.g., an API call 1544) to a server, such as an annotation assistant server 1540, which may contain a set of pre-trained models 1542 stored, for example, in an annotation model register. In at least one embodiment, an annotation model register can store pre-trained models 1542 (e.g., machine learning models, such as deep learning models) that are pre-trained to perform AI-assisted annotation on a specific organ or anomaly. These models can be further updated using training pipelines. In at least one embodiment, pre-installed annotation tools can be improved over time as new labeled data is added.

[0146] Different embodiments can be described by the following sentences: 1. Computer-implemented method, including: Generating a first noisy image and a second noisy image using forward diffusion and at least based on a reference image; Generating an upsampled image that corresponds to a scene represented by the reference image; Generating an output image that corresponds to the scene, at least on the basis of the upsampled image, the second noisy image, and a historical image; Generating a noisy output image, at least based on the output image; and Determining a loss parameter for training a generator, at least on the basis of a comparison between the second noisy image, the upsampled image, the historical image and the first noisy image and / or the noisy output image. 2. Computer-implemented method according to sentence 1, wherein the reference image is a high-resolution image that has undergone downsampling. 3. Computer-implemented method according to sentence 2, which further includes: Receiving an image of the scene from an engine with a first resolution; and Reducing the image from the first resolution to a lower second resolution to create the reference image. 4. Computer-implemented method according to sentence 1, which further comprises: Determining a second loss for training a discriminator based at least on the comparison between the second noisy image, the upsampled image, the historical image and the first noisy image and / or the noisy output image. 5. Computer-implemented method according to sentence 1, wherein the historical image is at least one of: a first image in a sequence of images, the upsampled image, or the output image from a previous iteration. 6. Computer-implemented method according to sentence 1, which further comprises: Updating a trained generator using a set of weights based on loss; Forming a content generation pipeline that includes the trained generator and a rendering engine that provides input images to the trained generator; and Generating an output sequence according to the input images from the content generation pipeline. 7. Computer-implemented method according to sentence 1, which further comprises: Determine that a certain number of iterations is below a threshold; and Generating a second output image. 8. Computer-implemented method according to sentence 7, which further includes: Determine that the number of iterations reaches a threshold; and Receiving a second reference image. 9. Processor comprehensive: one or more circuits: to generate a series of noisy images based on a reference image; to select a final noisy image from the series of noisy images; to provide the final noisy image, a low-resolution upsampled image, and a historical image to a generator to produce an output image; to generate a noisy output image based at least on the initial image produced using the generator; and to adjust one or more parameters of the generator based on a comparison between the final noisy image, the upsampled low-resolution image, the historical image and the noisy output image and / or a noisy intermediate image from the series of noisy images. 10. Processor according to Theorem 9, wherein the comparison is based on the noisy output image for a first pass and on the noisy intermediate image for a second pass. 11. Processor according to sentence 9, wherein the historical image corresponds to the upsampled, low-resolution image. 12. Processor according to sentence 9, wherein one or more circuits are further designed: to receive an output from an engine with a first resolution; and to generate the upsampled, low-resolution output from the engine output at a second resolution. 13. Processor according to sentence 9, wherein one or more circuits are further designed: to create a set of weights for the generator. 14. Processor according to sentence 9, wherein the processor comprises at least one of: a system for performing simulation operations; a system for performing simulation operations to test or validate autonomous machine applications; a system for performing digital twin operations; a system for performing light transport simulations; a system for rendering graphical output; a system for performing deep learning operations; a system implemented using an edge facility; a system for generating or displaying virtual reality (VR) content; a system for generating or displaying augmented reality content or AR content; a system for generating or displaying mixed reality or MR content; a system that contains one or more virtual machines or VMs; a system for performing operations for a conversational AI application; a system for performing operations for a generative AI application; a system for performing operations using a language model; a system for performing one or more generative operations on content using a large language model or LLM; a system for performing one or more generative operations using a vision language model or VLM; a system that is at least partially implemented in a data center; a system for performing hardware tests using simulations; a system for performing one or more generative operations using a language model; a system for generating synthetic data; a platform for the collaborative creation of content for 3D assets; or a system that is implemented at least partially using cloud computing resources. 15. System comprehensive: one or more processing units to generate an output image from a multitude of input images using a trained generator, wherein the multitude of input images includes at least one noisy image, which is generated by adding noise to a reference image of a scene, contains a low-resolution upsampled image of the scene and a historical image of the scene. 16. System according to Theorem 15, wherein the historical image of the scene is extracted from a buffer after at least one pass of the trained generator. 17. System according to sentence 16, wherein the one or more processing units are further configured to iteratively generate the output image after a threshold number of iterations has been performed. 18. System according to sentence 15, wherein the one or more processing units are further configured to modify an output image of an engine with a first resolution in order to produce the upsampled low-resolution image of the scene with a second resolution. 19. System according to sentence 15, wherein the output image is at least one of: an image with a higher resolution than the plurality of input images or an image that is a duplicate of one of the plurality of input images. 20. System according to sentence 15, where the system is one of: a system for performing simulation operations; a system for performing simulation operations to test or validate autonomous machine applications; a system for performing digital twin operations; a system for performing light transport simulations; a system for rendering graphical output; a system for performing deep learning operations; a system implemented using an edge facility; a system for generating or displaying virtual reality (VR) content; a system for generating or displaying augmented reality content or AR content; a system for generating or displaying mixed reality or MR content; a system containing one or more virtual machines (VMs); a system for performing operations for a conversational AI application; a system for performing operations for a generative AI application; a system for performing operations using a language model; a system for performing one or more generative operations on content using a large language model or LLM; a system for performing one or more generative operations using a vision language model or VLM; a system that is at least partially implemented in a data center; a system for performing hardware tests using simulations; a system for performing one or more generative operations using a language model; a system for generating synthetic data; a platform for the collaborative creation of content for 3D assets; or a system that is implemented at least partially using cloud computing resources.

[0147] Other variations are within the spirit of the present disclosure. Thus, while various modifications and alternative designs can be made with respect to the disclosed methods, certain illustrated embodiments are shown in the drawings and have been described in detail above. However, it is understood that the intention is not to limit the disclosure to the specific disclosed form or forms, but rather, on the contrary, to cover all modifications, alternative designs, and equivalents that fall within the spirit and scope of the disclosure as defined in the attached claims.

[0148] The use of the terms "a," "an," "the," and similar referents in the context of describing disclosed embodiments (particularly in the context of the following claims) is to be interpreted as covering both the singular and the plural unless otherwise specified herein or the context clearly contradicts this, and not as defining an expression. The terms "comprising," "having," "including," and "containing" are to be interpreted as open expressions (i.e., in the sense of "including without being limited to") unless otherwise specified. The term "connected" is to be interpreted as partially or completely contained within, attached to, or joined to one another when used unmodified and referring to physical connections, even if an element is inserted between them.The mention of ranges of values ​​herein is intended merely as a quick method of referring individually to each separate value falling within the range, unless otherwise stated herein, and each separate value is included in the description as if it were reproduced individually herein. The use of the expression "sentence" (e.g., "a set of objects") or "substance" is to be understood as a non-empty compilation comprising one or more elements, unless otherwise noted or the context contradicts it. Furthermore, the term "subset" of a corresponding sentence, unless otherwise stated or the context contradicts it, does not necessarily mean a proper subset of the corresponding sentence; rather, the subset and the corresponding sentence may be the same.

[0149] Unless specifically stated otherwise or the context clearly contradicts it, connective language, such as phrases of the form "at least one of A, B, and C" or "at least one of A, B, and C," is otherwise to be understood in the context in which it is generally used to indicate that an object, expression, etc., can be either A, B, or C, or any non-empty subset of the sentence consisting of A, B, and C. For example, in the illustrated example of a sentence containing three elements, the connective phrases "at least one of A, B, and C" and "at least one of A, B, and C" refer to any one of the following: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}.Thus, such connecting expressions are generally not intended to express that certain executions require the presence of at least one of A, at least one of B, and at least one of C. Additionally, unless otherwise stated or contradicted by the context, the term "multiplicity" also denotes a state of plurality (e.g., "a multitude of elements" denotes multiple elements). A multitude consists of at least two elements, but may also include more if this is either explicitly stated or indicated by the context. Furthermore, unless otherwise stated or evident from the context, the phrase "based on" means "at least partially based on" and not "exclusively based on."

[0150] The operations of processes described herein may be performed in any suitable order, unless otherwise specified herein or the context clearly precludes it. In at least one embodiment, a process, such as the processes described herein (or variations and / or combinations thereof), is carried out under the control of one or more computer systems configured with executable instructions, and is implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors, by hardware or combinations thereof. In at least one embodiment, code is stored on a computer-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions that can be executed by one or more processors.In at least one embodiment, a computer-readable storage medium is a non-transitory computer-readable storage medium that excludes transitory signals (e.g., a propagating transient electrical or electromagnetic transmission) but includes non-transitory data storage circuits (e.g., buffers, caches, and queues) within transient signal senders / receivers. In at least one embodiment, the code (e.g., executable code or source code) is stored on a set of one or more non-transitory computer-readable storage media containing executable instructions (or other storage for executable instructions) which, when executed (i.e., as a result of execution) by one or more processors of a computer system, cause the computer system to perform the operations described herein.In at least one embodiment, a set of nontransitory computer-readable storage media comprises multiple nontransitory computer-readable storage media. One or more of the individual nontransitory storage media do not contain the entire code, while multiple nontransitory computer-readable storage media collectively store the entire code. In at least one embodiment, the executable instructions are executed such that different instructions are executed by different processors—for example, a nontransitory computer-readable storage medium stores instructions, and a central processing unit (CPU) executes some of the instructions, while a graphics processing unit (GPU) executes other instructions.In at least one embodiment, different components of a computer system have separate processors, and different processors execute different subsets of instructions.

[0151] Accordingly, computer systems in at least one embodiment are configured to implement one or more services that, individually or collectively, perform operations of the processes described herein, and such computer systems are configured with applicable hardware and / or software that enables the execution of operations. Furthermore, a computer system implementing at least one embodiment of the present disclosure is a single device, and in another embodiment, a distributed computer system comprising several devices that operate differently, such that the distributed computer system performs the operations described herein and such that a single device does not perform all operations.

[0152] The use of any and all examples or illustrative language (e.g., "such as") provided in this document is intended solely to better clarify the embodiments of the disclosure and does not constitute a limitation of the scope of the disclosure unless claimed otherwise. No wording in the description should be interpreted as indicating any unclaimed element as essential to the implementation of the disclosure.

[0153] Any references, including publications, patent applications and patents mentioned in this document, are hereby incorporated by reference to the same extent as if each reference had been individually and specifically indicated as being included by reference and set forth in this document in its entirety.

[0154] In the description and claims, the terms "coupled" and "connected," along with their derivatives, may be used. It is understood that these terms are not intended to be synonymous. Rather, in specific examples, "connected" or "coupled" can be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. "Coupled" can also mean that two or more elements are not in direct contact with each other, but nevertheless interact or work together.

[0155] Unless expressly stated otherwise, terms such as "processing", "calculating", "calculating", "determining" or the like throughout this description are understood to refer to actions and / or processes of a computer or computing system or similar electronic computing device that manipulate and / or convert data represented as physical, e.g. electronic, quantities in the registers and / or memory of the computing system into other data represented in a similar manner as physical quantities in the memory, registers or other such information storage, transmission or display devices of the computing system.

[0156] Similarly, the term "processor" can refer to any device or section of a device that processes electronic data from registers and / or memories and converts that electronic data into other electronic data that can be stored in registers and / or memories. As non-restrictive examples, the "processor" can be a CPU or a GPU. A "computing platform" can include one or more processors. As used herein, "software" processes can include, for example, software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Furthermore, each process can refer to multiple processes for executing instructions sequentially or in parallel, continuously or intermittently.The terms “system” and “procedure” are used interchangeably in this document insofar as a system can embody one or more procedures and the procedures can be considered as a system.

[0157] This document may refer to the acquisition, capture, reception, or input of analog or digital data into a subsystem, computer system, or computer-implemented machine. The acquisition, capture, reception, or input of analog and digital data can occur in various ways, such as receiving data as a parameter of a function call or a call to an application programming interface. In some implementations, the process of acquiring, capturing, receiving, or inputting analog or digital data can be achieved by transmitting data over a serial or parallel interface. In another implementation, processes of acquiring, capturing, receiving, or inputting analog or digital data can be achieved by transmitting data over a computer network from the providing entity to the receiving entity.It can also refer to the provision, output, transmission, sending, or representation of analog or digital data. In various examples, the process of providing, outputting, transmission, sending, or representing analog or digital data can be accomplished by transmitting data as input or output parameters of a function call, a parameter of an application programming interface, or an interprocess communication mechanism.

[0158] Although the preceding discussion presents exemplary implementations of the described techniques, other architectures may also be used to implement the described functionality, and they are intended to be within the scope of this disclosure. Furthermore, although specific distributions of responsibilities are defined above for the purpose of discussion, various functions and responsibilities could be distributed and divided differently depending on the circumstances.

[0159] Although the subject matter has been further described in language specific to structural features and / or process steps, it is understood that the subject matter claimed in the appended claims is not necessarily limited to the specific features or steps described. Rather, specific features and steps are disclosed as exemplary ways of implementing the claims.

Claims

[1] Computer-implemented method, comprising: Generating a first noisy image and a second noisy image using forward diffusion and at least based on a reference image; Generating an upsampled image that corresponds to a scene represented by the reference image; Generating an output image that corresponds to the scene, at least on the basis of the upsampled image, the second noisy image, and a historical image; Generating a noisy output image, at least based on the output image; and Determining a loss parameter for training a generator, at least on the basis of a comparison between the second noisy image, the upsampled image, the historical image and the first noisy image and / or the noisy output image. [2] Computer-implemented method according to claim 1, wherein the reference image is a high-resolution image that has undergone downsampling. [3] Computer-implemented method according to claim 2, further comprising: Receiving an image of the scene from an engine with a first resolution; and Reducing the image from the first resolution to a lower second resolution to create the reference image. [4] Computer-implemented method according to any of the preceding claims, further comprising: Determining a second loss for training a discriminator based at least on the comparison between the second noisy image, the upsampled image, the historical image and the first noisy image and / or the noisy output image. [5] Computer-implemented method according to any of the preceding claims, wherein the historical image is at least one of: a first image in a sequence of images, the image subjected to upsampling or the output image from a previous iteration. [6] Computer-implemented method according to any of the preceding claims, further comprising: Updating a trained generator using a set of weights based on loss; Forming a content generation pipeline that includes the trained generator and a rendering engine that provides input images to the trained generator; and Generating an output sequence according to the input images from the content generation pipeline. [7] Computer-implemented method according to any of the preceding claims, further comprising: Determine that a certain number of iterations is below a threshold; and Generating a second output image. [8] Computer-implemented method according to claim 7, further comprising: Determine that the number of iterations reaches a threshold; and Receiving a second reference image. [9] Processor including: one or more circuits: to generate a series of noisy images based on a reference image; to select a final noisy image from the series of noisy images; to provide the final noisy image, a low-resolution upsampled image, and a historical image to a generator to produce an output image; to generate a noisy output image based at least on the initial image produced using the generator; and to adjust one or more parameters of the generator based on a comparison between the final noisy image, the upsampled low-resolution image, the historical image and the noisy output image and / or a noisy intermediate image from the series of noisy images. [10] Processor according to claim 9, wherein the comparison is based on the noisy output image for a first pass and on the noisy intermediate image for a second pass. [11] Processor according to claim 9 or 10, wherein the historical image corresponds to the upsampled low-resolution image. [12] Processor according to one of claims 9 to 11, wherein one or more circuits are further configured as follows: to receive an output from an engine with a first resolution; and to generate the upsampled, low-resolution output from the engine output at a second resolution. [13] Processor according to one of claims 9 to 12, wherein one or more circuits are further configured as follows: to create a set of weights for the generator. [14] Processor according to any one of claims 9 to 13, wherein the processor comprises at least one of: a system for performing simulation operations; a system for performing simulation operations to test or validate autonomous machine applications; a system for performing digital twin operations; a system for performing light transport simulations; a system for rendering graphical output; a system for performing deep learning operations; a system implemented using an edge facility; a system for generating or displaying virtual reality (VR) content; a system for generating or displaying augmented reality content or AR content; a system for generating or displaying mixed reality or MR content; a system that contains one or more virtual machines or VMs; a system for performing operations for a conversational AI application; a system for performing operations for a generative AI application; a system for performing operations using a language model; a system for performing one or more generative operations on content using a large language model or LLM; a system for performing one or more generative operations using a vision language model or VLM; a system that is at least partially implemented in a data center; a system for performing hardware tests using simulations; a system for performing one or more generative operations using a language model; a system for generating synthetic data; a platform for the collaborative creation of content for 3D assets; or a system that is implemented at least partially using cloud computing resources. [15] System encompassing: one or more processing units to generate an output image from a multitude of input images using a trained generator, wherein the multitude of input images includes at least one noisy image, which is generated by adding noise to a reference image of a scene, contains a low-resolution upsampled image of the scene and a historical image of the scene. [16] System according to claim 15, wherein the historical image of the scene is extracted from a buffer after at least one pass of the trained generator. [17] System according to claim 16, wherein the one or more processing units are further configured to iteratively generate the output image after a threshold number of iterations has been performed. [18] System according to one of claims 15 to 17, wherein the one or more processing units are further configured to modify an output image of an engine with a first resolution in order to generate the upsampled low-resolution image of the scene with a second resolution. [19] System according to any one of claims 15 to 18, wherein the output image is at least one of: an image with a higher resolution than the plurality of input images or an image that is a duplicate of one of the plurality of input images. [20] System according to any one of claims 15 to 19, wherein the system is one of: a system for performing simulation operations; a system for performing simulation operations to test or validate autonomous machine applications; a system for performing digital twin operations; a system for performing light transport simulations; a system for rendering graphical output; a system for performing deep learning operations; a system implemented using an edge facility; a system for generating or displaying virtual reality (VR) content; a system for generating or displaying augmented reality content or AR content; a system for generating or displaying mixed reality or MR content; a system that contains one or more virtual machines or VMs; a system for performing operations for a conversational AI application; a system for performing operations for a generative AI application; a system for performing operations using a language model; a system for performing one or more generative operations on content using a large language model or LLM; a system for performing one or more generative operations using a vision language model or VLM; a system that is at least partially implemented in a data center; a system for performing hardware tests using simulations; a system for performing one or more generative operations using a language model; a system for generating synthetic data; a platform for the collaborative creation of content for 3D assets; or a system that is implemented at least partially using cloud computing resources.

Images