Immersive animated content made from static images using generative artificial intelligence

Abstract

The approaches presented here can be used to generate animated images using static two-dimensional images (static 2D images). An input image can be used to generate a first image corresponding to the foreground of the input image and at least one second image corresponding to the background of the input image. The second image can include an inpainted area based on a mask generated from the first image. The first and second images can be provided with configuration settings for rendering on a client device.

Description

BACKGROUND

[0001] Users requesting content and / or access to resources may experience wait times while resources are procured and / or content is identified and delivered to the user. During this wait, the user may be presented with information indicating that the request is pending, instead of a blank screen or image. This can be confusing for the user, as it may mean that the request was not received, the screen is frozen, or something similar. Static images can be used to reduce computational overhead, but they can provide a less immersive experience for users. Additionally, a consistently static image can cause the same problems as a blank screen, namely confusion if the user receives no indication of the status of their request.It may be preferable to provide an improved wait experience for a user, making them more likely to use the service in the future. However, generating specific content solely for wait times can be costly and may increase wait times due to bandwidth usage, network connectivity, and / or limited computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Various embodiments according to the present disclosure are described with reference to the drawings, in which: Fig. 1A illustrates an exemplary environment for content generation in a provider environment according to different embodiments; Fig. 1B illustrates an exemplary environment for a content generation engine according to different embodiments; Fig. 2A illustrates an exemplary environment for retrieving an input image according to different embodiments; Fig. 2B illustrates an exemplary environment for extracting a foreground area from an image according to various embodiments; Fig. 2C illustrates an exemplary environment for generating a mask of a foreground image in an image according to various embodiments; Fig. 2D illustrates an exemplary environment for inpainting within a mask area according to various embodiments; Fig. 2E illustrates an exemplary environment for outpainting along a border area of ​​an image according to various embodiments; Fig. 2F illustrates an exemplary environment for generating a layered output configuration according to various embodiments; Fig. 3 An exemplary call diagram for generating animation configuration files and providing animation configuration files to a user is illustrated according to different embodiments; Fig. 4A illustrates an exemplary process for generating an animation effect according to different embodiments; Fig. 4B illustrates an exemplary process for selecting and providing an animation effect according to different embodiments; Fig. Figure 4C illustrates an exemplary process for generating an animation effect according to different embodiments; Fig. 5 illustrates an exemplary process for generating a configuration file for running an animation using two or more images according to different embodiments; Fig. 6 components of a distributed system illustrated that can be used to update or perform inference using a machine learning model according to at least one embodiment; Fig. 7A Inference and / or training logic illustrated according to at least one embodiment; Fig. 7B Inference and / or training logic illustrated according to at least one embodiment; Fig. 8 illustrates an exemplary data center according to at least one embodiment; Fig. 9 illustrates a computer system according to at least one embodiment; Fig. 10 illustrates a computer system according to at least one embodiment; Fig. 11 at least sections of a graphics processor according to one or more embodiments are illustrated; Fig. 12 at least sections of a graphics processor according to one or more embodiments are illustrated; Fig. 13 an exemplary data flow diagram for an advanced computing pipeline according to at least one embodiment; Fig. 14 a system diagram for an exemplary system for training, adapting, instantiating and deploying machine learning models in an advanced computing pipeline according to at least one embodiment; and The Fig. 15A and Fig. 15B illustrate a data flow diagram for a process for training a machine learning model and a client-server architecture for improving annotation tools with pre-trained annotation models according to at least one embodiment. DETAILED DESCRIPTION

[0003] The following description details various embodiments. For explanatory purposes, specific configurations and details are presented to provide a thorough understanding of the embodiments. However, it is clear to the person skilled in the art that the embodiments can be implemented without these specific details. Moreover, well-known features may be omitted or simplified so as not to obscure the described embodiment.

[0004] The systems and methods described herein can be used, for example, without restriction, by non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in a passenger compartment infotainment or digital or virtual driver assistance application), autonomous vehicles or machines, steered and unsteered robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled with one or more trailers, hydrofoils, boats, shuttles, emergency vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, underwater vehicles, drones and / or other types of vehicles.Furthermore, the systems and methods described herein can be used for a variety of purposes, including but not limited to machine control, machine locomotion, machine propulsion, synthetic data generation, model training or updating, perception, augmented reality, virtual reality, mixed reality, robotics, security and surveillance, simulation and digital twinning, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actuator simulation and / or digital twinning, data center processing, conversational artificial intelligence (conversational AI), generative AI with large language models (LLMs) and vision language models (VLMs), light transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation for 3D assets, cloud computing, and / or other suitable applications.

[0005] The disclosed embodiments can consist of a variety of different systems, such as automotive systems (e.g., a control system for an autonomous or semi-autonomous machine, a perception system for an autonomous or semi-autonomous machine), systems implemented using a robot, aviation systems, media systems, boating systems, intelligent area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twinning operations, systems implemented using an edge device, systems that include one or more virtual machines (VMs), systems for performing operations to generate synthetic data, systems that are at least partially implemented in a data center, and systems for performing conversational AI operations.Systems for performing generative AI operations, systems for performing operations that use LLMs and / or VLMs, systems for performing light transport simulation, systems for performing collaborative content creation for 3D assets, systems that are implemented at least partially using cloud computing resources, and / or other types of systems.

[0006] Approaches according to various embodiments can be used to generate one or more parameters for a content generation environment. In at least one embodiment, a trained machine learning (ML) system and / or an artificial intelligence (AI) system, such as a large language model (LLM) or a vision language model (VLM), can be used to generate parameters for the content generation environment, such as, but not limited to, camera settings, scene lighting, video parameters, and / or the like, used to display objects within a scene. The parameters can be based on input provided by a user or a user proxy to a trained language model (e.g., LLM, VLM, etc.), which can then generate one or more settings according to the input.Various embodiments can be used to generate settings in two-dimensional (2D) or three-dimensional (3D) environments. For embodiments that include one or more language models—that is, one or more LLMs, one or more VLMs, or a combination of LLMs and VLMs—the language model(s) can receive input (e.g., a prompt, a request, a query, etc.) that is parsed or otherwise formatted to produce deterministic output. The input provided to the language model can include, for example, a specific format for the output results, an example of undesired output results, a specific list of parameters and their respective formatting, and the like. An input generator (e.g.,An input prompt generator, driven or otherwise directed by one or more AI and / or ML systems, can be used to generate this input based on initial input received from a user, device, proxy, and / or the like. A modified input generated by the input generator can then be provided to the language model, which produces an output set of parameters. This output can further be evaluated by a reviewer or other system to ensure its suitability. Subsequently, a configuration file can be generated, and / or the parameters can be directly provided to an environment to configure various components (e.g., camera settings, lighting, etc.) based on the parameters generated by the language model.

[0007] Several embodiments of the present disclosure relate to generating immersive experiences for users when they would otherwise encounter static screens, such as, among other options, during screen loading or for a display. Implementations use one or more trained neural networks to transform an input image into a number (e.g., two, three, four, etc.) of separate images that can be used to generate animated effects presented to users instead of static images. In an exemplary implementation with three separate images, an initial input image (e.g., a two-dimensional image, such as a "hero image" from a video game, a sales item, or a user-selected image) can be processed to identify and extract the foreground.The extracted foreground components, or at least some of them, can be identified and then stored as a new image. In at least one embodiment, a mask can be created that corresponds to the extracted foreground components using the original image. Then, one or more trained neural networks can be used for inpainting to fill the mask by creating a second image. Additionally, a third image can be generated by one or more neural networks for outpainting the image. The collection of multiple images (e.g., foreground, inpainted background, outpainted background, other objects derived from the scene, etc.) can then be used to generate one or more effects such as panning, zooming, and / or the like.These images can be processed offline and then provided to a user device for rendering at runtime. This allows for an enhanced and more interactive user experience.

[0008] The systems and methods disclosed herein can be used to address and overcome problems with existing systems where a static image may be presented to a user. This image may have limited interactivity and / or may be boring or misleading for the user. For example, a static image may eventually lead a user to mistakenly believe that their request is "frozen" or has not been processed. Additionally, a user may become tired of the static image and choose to view a different screen or use a completely different service.Embodiments of the present disclosure can solve these problems by using, in at least one embodiment, one or more trained machine learning models for foreground / background separation and / or inpainting / outpainting to create one or more immersive effects. Such embodiments can be used when there is a "waiting time" for one or more operations or services, such as loading a screen for a game browser, loading a screen for a specific game, a shopping environment, requesting resources, and / or the like.

[0009] In at least one embodiment, systems and methods can be used to engage the user by providing an immersive or eye-catching way to interact with content (e.g., game content, shop window content, etc.) while the user is waiting for an action and / or before the user performs an action. For example, the user might be waiting for an application to load or for resources to be procured and become available. As another example, a user might be visiting an online shop window where static images of potential items for sale are presented. Previous solutions display static images or videos that are rarely, if ever, updated and / or whose use and rendering are costly.Consequently, at least part of the user interface (UI) for a given application or action is outdated, occupying space that could otherwise be used to engage or otherwise delight the user. Adding effects to at least part of this space can improve the user experience and / or draw the user's attention to a specific area, which can increase enthusiasm for and / or usage of the service or storefront.

[0010] Embodiments can be used to generate several different types of content, for example, by using different static images to create immersive content. Content can be user-specific (e.g., based on user preferences provided by the user, etc.) and / or generic (e.g., based on content or platforms). In at least one embodiment, static images are made interactive by using one or more generative artificial intelligence (AI) techniques and then providing the generated content to a client device for rendering. Content can be personalized for a given user, such as by incorporating images provided by the user or by involving the user in a game environment.Additionally, implementations can be used to combine immersive content with informative content, such as a loading bar or expected wait time, to further enhance the user experience. Interactivity can also be provided by offering users input options with the generated content, such as tracking input movements. Consequently, a user can experience an improved waiting experience.

[0011] Various other such functions can also be used within the scope of protection of the various embodiments, as is obvious to the person skilled in the art in view of the teachings and proposals contained herein.

[0012] The invention is defined by the claims. To illustrate the invention, aspects and embodiments that may or may not fall within the scope of the claims may be described herein.

[0013] This paper presents approaches for generating animated images using static two-dimensional images (static 2D images). An input image can be used to generate a first image, corresponding to the foreground of the input image, and at least one second image, corresponding to the background of the input image. The second image can include an inpainted area based on a mask generated from the first image. The first and second images can be provided with configuration settings for rendering on a client device.

[0014] The revelation extends to every new aspect and / or feature described and / or illustrated herein. Further features of the revelation are characterized by the independent and dependent claims.

[0015] Any feature of one aspect of the disclosure can be applied to other aspects of the disclosure in any suitable combination. In particular, procedural aspects can be applied to device or system aspects and vice versa.

[0016] Furthermore, features implemented in hardware can be implemented in software and vice versa. Any reference to software and hardware features herein should be interpreted accordingly.

[0017] Each system or device feature as described herein can also be provided as a process feature, and vice versa. System and / or device aspects described functionally (including means plus functional features) can alternatively be expressed in terms of their corresponding structure, such as a suitably programmed processor and allocated working memory.

[0018] It is also understood that special combinations of the various features described and defined in aspects of the revelation can be implemented and / or supplied and / or used independently.

[0019] The disclosure also provides computer programs and computer program products comprising software code which, when executed on a data processing device, is adapted to perform one of the procedures and / or to embody one of the device and system features described herein, including all component steps of any procedure.

[0020] The disclosure also provides a computer or computing system (including networked or distributed systems) comprising an operating system that provides a computer program for performing the procedures described herein and / or for embodying any device or system features described herein.

[0021] The revelation also provides a computer-readable medium on which one or more of the aforementioned computer programs are stored.

[0022] The revelation also provides a signal that carries one or more of the aforementioned computer programs.

[0023] The disclosure extends to methods and / or devices and / or systems as described herein with reference to the accompanying drawings.

[0024] Aspects and embodiments of the present disclosure will now be described purely by way of example with reference to the accompanying drawings.

[0025] Fig. Figure 1 illustrates an exemplary environment 100 that can be used in embodiments of the present disclosure. In this example, a user device 102 can request the use of one or more resources 104 and / or access content from a content data store 106 from a provider environment 108. The user device 102 can, for example, be a proxy for one or more users and / or automated workflows that cause or instruct the user device 102 to submit requests to the provider environment 108 over one or more networks. In at least one embodiment, the user device 102 can be any suitable computing or processing device, such as a desktop or notebook computer, a smartphone, a tablet computer, a portable computer (e.g., a smartwatch, eyeglasses, contact lenses, a headset, etc.).), includes a server or other such system or device. Additionally, the one or more networks may include any suitable network, such as the Internet, a Local Area Network (LAN), a mobile network, Ethernet, or any other such wired and / or wireless network.

[0026] In at least one embodiment, the provider environment 108 can grant access to one or more resources 104 and / or content in response to a determination that the user device 102 is authorized to access the resources 104 and / or content. The provider environment 108 can, for example, be associated with a distributed computing environment (e.g., the "cloud") that allows users to access different resources and / or content, which may include resources or content for running one or more applications remotely from the user device itself. The resources 104 may, for example, include computing resources such as central processing units (CPUs) and / or graphics processing units (GPUs), which may include hardware with more extensive capabilities than the hardware associated with the user device 102.By using the resources 104 of the provider environment 108, the user of the user device 102 can gain access to a greater variety of content and / or applications due to the improved processing capabilities of the resources 104.

[0027] As a non-restrictive example, the provider environment 108 can be associated with a cloud gaming or entertainment service, such as GeForce NOW from Nvidia Corporation. The provider environment 108 can, for example, allow a user to access high-performance resources through a subscription service using one or more network connections. As a result, instead of purchasing the resources themselves, users can effectively rent or otherwise access the resources 104 using the provider environment 108. When a user makes a request, a manager 110 can determine an approval or access level for the requesting user, for example, by querying an account data store 112 to verify login credentials, access limits, and / or the like.Users can be granted "access levels" or tiers, with certain tiers providing resources with higher performance, access to a larger content library, reduced wait times, and / or combinations thereof. After verifying an access level, the manager (110) can make one or more calls to a provisioning engine (114) to provision the resources (104) in response to the request. The resources (104) might include, for example, one or more GPUs to run a gaming application using content from the content data store (106).

[0028] During operation, the provider environment can be used to serve hundreds or thousands of users simultaneously or semi-simultaneously, and consequently, there may be delays or latency associated with verifying account access, provisioning resources, and / or preparing applications to run on the provisioned resources. Additionally, a user's level or tier can cause wait times for that user. For example, a user might submit a request to run an application with significant processing requirements but only have access to a lower tier of hardware, and consequently, there may be latency or delays in the request. During these wait times, the user may often be presented with a loading or waiting screen.As explained herein, loading screens may include static images or repeating videos that are rarely updated, thus occupying screen space without providing any value to the user. Embodiments of the present disclosure address and overcome these problems by employing one or more content generation engines 116 to generate an animated set of images from one or more static images, which can be used to provide users with a more immersive experience during the loading period. In at least one embodiment, the one or more content generation engines 116 can receive one or more still images, for example, from the content data store 106, extract foreground and background sections, perform inpainting, and then initiate one or more animations at the user device 102.In this way, the loading time for the user can be more immersive compared to conventional static or repeatedly looped images.

[0029] Fig. Figure 1B illustrates an exemplary environment 120 that can be used in embodiments of the present disclosure. In this example, one or more features of the content generation engine 116 are illustrated, which may include one or more trained machine learning systems, such as neural networks, that can be used for object identification, object extraction, segmentation, masking, inpainting, outpainting, and / or combinations thereof. Systems and methods may include one or more content generation pipelines that can be used to generate one or more images from an initial input image and then produce an output configuration that can be used to generate an effect or animation to be rendered on one or more client devices.As explained herein, systems and procedures can be directed toward pre-generated output configurations or output configurations that are generated in response to one or more user inputs or requests associated with a provider environment.

[0030] The content generation pipeline can include one or more machine learning systems and can, among other options, generate one or more intermediate images from an initial input or base image. In at least one embodiment, generated intermediate images can be provided as part of the output configuration or used to generate one or more additional images. For example, the pipeline can include an input image provided to the content generation engine 116. The input image can be a high-resolution image associated with a content element, an image selected by a user, a randomly selected image, and / or combinations thereof. Randomly selected high-resolution images, for instance, can be used to generate different varieties of output animations for different content.While high-resolution images are described as an example, embodiments can also be used with standard-resolution images, with still images extracted from videos, with models, and / or combinations thereof. In at least one embodiment, a processing engine 122 (e.g., a preprocessing engine) can be used to receive and prepare the input image for use with the pipeline, such as by checking the image format, cropping or rotating the image, setting one or more image parameters, and / or combinations thereof. The processing engine 122 can include a variety of preprocessing steps to prepare the image according to one or more settings, such as a desired output size, a desired resolution, and / or combinations thereof.

[0031] An extraction engine 124 can be used to extract one or more areas of an image, such as extracting content elements or foreground areas from background areas. One or more deep learning or instance / semantic segmentation networks can be used to identify and extract foreground areas of the image, which may be based on depth information, continuous object detection, specific object recognition, and / or combinations thereof. The extracted foreground portion can then be stored as a new image, for example, an image that includes only the extracted foreground portion. Additionally, in various implementations, further objects related to the foreground object can also be extracted.As an example, in a scene from a football match, a player might be running down a field and can be extracted as a foreground object, and a related object might be a football being passed to the player. Accordingly, each of these objects can be identified and extracted as one or both of the foreground image and / or related object image, which can then be used to animate or otherwise deliver a more immersive experience to a wait screen, as explained herein.

[0032] Embodiments can further process the image by generating a mask corresponding to the extracted foreground region, for example, using a masking engine 126. In certain embodiments, the masking engine 126 and the extraction engine 124 can use the same neural network architecture and / or can be different outputs generated by a common network. In other embodiments, however, different networks can be used. The mask can be converted into a monochrome image in certain embodiments.

[0033] One or more embodiments may also include an inpainting engine 128, which may be a prompt-based trained machine learning system, such as a diffusion model, that can be used to generate content within the mask area. In certain embodiments, the prompt is based at least partially on the context of the input image. If the input image belongs to a first-person shooter, the prompt information may include the game's context, an environment in which the game takes place (for example, outer space, World War II, etc.), and / or user preferences. The prompt may also be based on features of the extracted foreground, such as objects detected in the foreground, which can then be used as an example to formulate the prompt for the diffusion model.If the foreground is to be used to extract a soccer player, the prompt can include an identification for the sport (e.g., soccer) associated with the content, which can then provide information to the diffusion model to generate content that can be used for inpainting one or more areas, such as within the mask area. In at least one embodiment, the prompt can be automatically generated using one or more machine learning systems, such as a VLM, that can receive the input image and / or the extracted foreground and then generate a caption to describe the image's content. Additionally, prompts can be tailored and / or otherwise customized for specific use cases and / or image context.For example, prompts for sports games might follow one specific format, while prompts for first-person shooters follow another, and prompts for content elements in a storefront follow a different format based on type. Inpainting within the mask area can create a new image that can correspond to a new background image. Inpainting can be used, for example, to replace pixels that were previously "covered up" or otherwise adjusted to represent the extracted foreground image. Therefore, the initial input image can be used to create two output images at this point in the pipeline: one foreground (e.g., the extracted section) and one background (e.g., the background with the inpainted mask area).

[0034] Various embodiments can include an outpainting engine 130, which can also be a prompt-based trained machine learning system, such as a diffusion model. In at least one embodiment, each of the inpainting engine 128 and the outpainting engine 130 can share different layers or architectures or be a common model. As explained herein, outpainting can refer to generating a larger canvas or size for an image and can include adding pixels to a border region of an input image. The output engine 130 can also receive a prompt to instruct the diffusion model or other content generation model to generate a new output image. As a result, this exemplary pipeline can produce a total of three output images (e.g.,Foreground, background and new larger canvas), but different embodiments may include additional images, and three output images are provided as a non-restrictive example.

[0035] The images can then be provided to an output content generator 132, which can provide one or more output configurations to be rendered by an end user on a client device. For example, one or more animation effects can be applied to images, such as those generated as a result of the pipeline, to provide a more immersive effect and environment than a static image. For example, effects such as parallax, zooming, panning, and / or combinations thereof can be used to prepare the output configuration for use and rendering on the client device. In this way, an initial input image can be processed by the pipeline to produce an immersive, animated output configuration.Output configuration 132 can be provided as JSON or another set of instructions that can identify specific effects, effect parameters, and / or images for use in the effects. A variety of different effects can be generated using the same images, such as one zoom effect and another pan effect. One or more content generation pipelines can include instructions for generating different types of output configurations, based on properties of image usage, application context, and / or combinations thereof.

[0036] The systems and methods disclosed herein can be used to generate immersive content from one or more initial static images associated with a content element, such as, among other options, a single frame from a game. The systems and methods can process an initial input image, extract a foreground (e.g., one or more objects within the image, portions of the foreground from the image, one or more associated objects, etc.), inpaint a background area associated with a mask, output a larger canvas, and then merge the foreground and background. This approach can allow the foreground to be animated separately from the background to provide more immersive three-dimensional (3D) effects, even when starting from only static images.Accordingly, various implementations can utilize one or more machine learning systems, such as generative AI-based content generation models, to create more immersive content displays that include animation mechanisms applied to original static content. These implementations address problems with simple background images, such as during wait or loading periods when a user requests content or resources that are not immersive and / or do not enhance the user experience with a product or service. While videos can be considered more immersive than images, animated videos are resource-intensive and can be difficult to run at high resolutions on client devices without sufficient hardware capabilities.Certain environments may be designed to run on thin clients, for example, when using distributed services that utilize other hardware resources to perform tasks, and where video decoding may be too resource-intensive. Implementations address this problem, among other things, by using a limited or target set of static images (e.g., two or more), with one or more animation effects, to provide an immersive, video-like appearance during wait times without the overhead associated with video decoding.

[0037] Various implementations provide a workflow or pipeline for generating immersive content with 3D effects from static images. Input to the workflow can include one or more images associated with a content request, such as a "hero image" for a game, an image of a product for sale in a marketplace, a user-selected image, an image captured during user activity related to the content or resources, and / or combinations thereof. A foreground / background extraction tool, which may include one or more trained machine learning systems, can be used to separate a main object (e.g., a foreground section) from the background of the image. Tools can also be used to identify and separate objects related to the main object.An example extraction tool might include an Inverse Saliency Pyramid Reconstruction Network (InSPyReNet), a saliency object capture framework that can be based on a Res2Net or Swin transformer backbone. The generated foreground object can be used to create a mask, and then, using an inpainting prompt, implementations can fill the mask area using one or more generative AI workflows. For example, a diffusion network can be used to create a filled background image. The same or a different model can then be used to perform an outpainting step using an outpainting prompt. The generated images can then be treated as separate layers, and an output configuration is created for rendering on the client device.The output configuration can include one or more effects, such as parallax, geometric 3D, panning, zooming, and / or the like. Similarly, multiple output configurations can be created and then played back during the wait time, either in a loop or randomly / semi-randomly selected. This can make the wait time more immersive for the user.

[0038] The Fig. Figures 2A-2F illustrate an exemplary sequence of a Pipeline 200 that can be used with embodiments of the present disclosure. In this example, the sequence may include steps such as obtaining an image, extracting a foreground, generating a mask, inpainting the mask, outpainting a background, and then generating an output configuration that includes layers for animating the foreground relative to the background. This pipeline, as well as others, may contain more or fewer steps, and additional images may be used and / or generated within the scope of the present disclosure.

[0039] Fig. Figure 2A illustrates a pipeline segment that retrieves an image 202 from content data store 106, for example, in response to a request. The request can be provided by one or more client devices (and, through their extensions, by one or more users) and / or can be part of an automated workflow. For example, if content has been associated with video games, a number of popular games can be identified, and then different animated configurations can be generated and stored for a subset of the list. As another example, content data store 106 can include content provided by a game publisher, such as a "hero image" or a high-resolution image.Additionally, the content data store can include 106 images and / or videos assigned to a user of the content, which can be used to generate various immersive animations for use during a wait screen. As explained herein, different embodiments can process images in real time or near real time and / or can process images offline and then store one or more images and / or intermediate frames.

[0040] The image 202 in this example includes a single frame from a video game, including a character 204 within a scene 206. The character 204 can be considered the "foreground" of the scene 206 because the character 204 may be the main focal point or most important part of the scene for a user. In certain embodiments, the image 202 may be processed with one or more processing steps, such as steps to enhance the color or resolution of the image 202.

[0041] Pipeline 200 continues in Fig. 2B by providing image 202 to extraction engine 124. As explained herein, extraction engine 124 can be used to identify object 204 within scene 206 and then extract object 204, which may correspond to the foreground, to generate a foreground image 208. The foreground image 208 may be based, at least in part, on associated components that are linked to object 204. In at least one embodiment, additional objects may also be linked to or otherwise associated with object 204, such as when object 204 is holding a weapon. In certain embodiments, object 204 may be automatically detected within scene 206. In other embodiments, however, object 204 may be specified or otherwise requested, such as when image 202 includes depth information or other distinguishing information that is linked to the foreground.The generated foreground image 208 can be stored in memory or saved in another way for use within pipeline 200. Additionally, the foreground image 208 can be stored separately for one or more additional downstream processes.

[0042] Fig. Figure 2C illustrates a masking process in which the foreground image 208 is processed by the masking engine 126 to form a masked image 210 that includes a masked region 212, which speaks to the object 204. In certain embodiments, a conversion step may also be included to generate the masked image 210. In this way, a region associated with the foreground can be identified within the initial image and, as shown in Figure 2C, Fig. The 2D example shows how the background can be inpainted to add additional information to the scene 206, for example, using one or more generative AI systems. In this example, the masked image 210 and a prompt to the inpainting engine 128 to produce a background image 214 are provided. The background image 214 can correspond to the background of the original image 202 and includes an inpainted area 216 that corresponds to the masked area 212. In other words, the inpainting engine 128 is used to fill in the content that is now available due to the removal of the object 204 associated with the masked area 212.

[0043] Fig. Figure 2E illustrates another section of the pipeline 200 by using one or more images from the outpainting engine 130 to generate a canvas image 218. In this example, one or both of the background image 214 or the image 202 can be used to generate the canvas image 218. Additionally, various other images within the scope of this disclosure can be used. Furthermore, a prompt, which may be a prompt provided to one or more diffusion models as explained herein, can also be used to generate the canvas image 218. In at least one embodiment, the prompt is associated with a context or theme of the image 202 among various other options, such as a "first-person shooter," a "football game," or a "brand wait screen."The prompt and image inputs can be used to generate canvas image 218, which can include additional pixels at different edges. Canvas image 218 shows added areas 220, 222; however, it is understood that there can be more or fewer added areas 220, 222, such as added areas along the top or bottom of canvas image 218. Compared to the initial image, more content is provided within the added areas 220, 222, which can be used when a zooming motion or other effect is added to the configuration described herein.

[0044] Fig. Figure 2F illustrates the generation of an output configuration 224 using the output content generation engine 132. In this example, a set of images 226 can include some or all of the images 202, the foreground image 206, the background image 214, and / or the canvas image 218, which can be used by the output content generation engine 126 to apply a layered set of images as part of the output configuration 224. This allows different areas or layers to move relative to each other by providing animation while using static images. In this example, the canvas image 218 provides the bottom layer, followed by the background image 214 and the foreground image 206.The images can be stacked or otherwise grouped within a target configuration, which can be a JSON configuration provided to a user device for rendering. In at least one embodiment, the JSON configuration can identify a target effect, including comments, provide layer identifications to select the desired images for the effect, and set layer or effect parameters (e.g., layer offsets, direction, duration, etc.). This allows the user to receive a more immersive experience while waiting on a wait screen, while also reducing computation and bandwidth usage allocated to video or processing-intensive wait screen images or content.In addition, embodiments of the present disclosure may also overlay additional information on the various images as other layers, such as a countdown time or other indicator for the user.

[0045] Implementations can also include an interactive experience associated with the configuration. For example, one or more properties of the output configuration can be set based on user interaction at render time. The interaction can take the form of user-provided input, such as mouse input, and can include modifying the animation speed, temporarily pausing an interaction, initiating the animation, and / or combinations thereof. As another example, the input can be received as an instruction to pan or rotate the output configuration. As a non-restrictive example, moving the mouse to the left could cause an animation to rotate to the left, moving the mouse to the right could cause a rotation to the right, and so on.Additionally, scrolling or other actions can be received as commands to zoom in or out or pan. This provides a more immersive experience because the user can control or otherwise influence the animated content on the waiting screen.

[0046] One or more embodiments can further utilize the configuration to initiate or otherwise enable a minigame or activity for the waiting user. Multiple layers can be used, for example, to generate the output configuration, which may include layers for a main object within the scene or for associated objects. In at least one embodiment, user input can be used to control movement of the main object or associated objects, among other options such as controlling a parallax effect or panning to align the associated objects with the destination. Accordingly, systems and methods can provide the user with an engaging activity while waiting, thereby increasing the user's enjoyment during a period when, in previous implementations, the user would otherwise have waited contemplating a static image.

[0047] Fig. Figure 3 illustrates an exemplary call diagram 300 that can be used in embodiments of the present disclosure. In this example, a series of calls, actions, and / or responses can illustrate the generation of one or more immersive, animated content elements that can be provided to a user during a waiting period, such as, among other options, a waiting period for an application or resources. In this example, the user device 102 can use one or more applications that can generate content 302, which can be stored in one or more content data stores 106. For example, the user device 102 can start and play a video game and can perform actions within the game that can be saved or recorded in some other way.User Device 102 can be used to configure one or more settings for recording or otherwise saving content, which may include saving content in response to a command, saving content at timed intervals, or not saving content, among other options and combinations thereof. User Device 102 can permit access to the content for additional supplementary content creation, as described herein.

[0048] In this example, Manager 110 can be used to transmit instructions for generating content, which can be used during wait times by submitting instructions 304 to the content generation engine 116. Instructions 304 can include a command or request to generate specific types of supplemental content, which can include parameters associated with the generated content. For example, the instructions can include a specific associated content element, a desired length, a desired output resolution, and / or combinations thereof. Additionally, the request can be provided offline (e.g., before the wait time) so that content can be generated and stored for later use upon request from a user device.A pre-built content set can therefore be generated and then delivered in response to a user request, which can save computational resources and / or reduce latency compared to generating content during operation or in real-time / near-real-time. In this example, 306 requests content from the content generation engine 116 from the one or more content data stores 106 in response to instructions received from the manager 110. The one or more content data stores 106 might include, for example, a "hero image" for a video game, user-generated content, images for a product for sale in a marketplace, and / or combinations thereof. The content generation engine 116 can then receive the content 308 and perform one or more content generation operations 310, which, as explained herein, might include generating an output configuration.The output configuration can then be transferred to the one or more content data stores 106 312. In at least one embodiment, the content generation engine 116 can provide information regarding the completed request back to the manager 110 314. As explained herein, instead of or in addition to providing the content to the one or more content data stores 106, the content generation engine 116 can also provide the content to the user device 102 if the content is produced in real time or near real time.

[0049] In at least one embodiment, the user device 102 can submit a request 316 to the manager 110 for access to one or more resources. The resources can be, for example, computing resources for running one or more applications, such as a video game. As another example, the request can involve access to a landing page for an online marketplace for purchasing one or more items, such as physical or electronic goods. The manager 110 can query an account data store 112 318 to obtain login credentials 320 in order to determine an access level permitted to the user device 102. Upon determining that the user device 102 is authorized to execute the request 316, the manager 110 can transmit instructions to the deployment engine 114 322 to procure / deploy one or more resources and / or applications 324.

[0050] As explained herein, a wait time associated with resources and / or application startup may exist. For example, if a user has a lower "level" for a distributed service, the user may wait longer than other users with higher levels. In certain implementations, if a user also requests a resource-intensive application, additional wait times may occur. While the user is waiting for resource utilization to complete, in certain implementations, the manager 110 may request one or more output content configurations from the one or more content data stores 106 326. The request 326 may include a specific selection for an output configuration, such as one associated with a particular application or resource.Additionally, request 326 can include multiple content requests if it is determined that the procurement / deployment 324 may exceed a threshold of time. After receiving 328 the output content configuration, the manager 110 can deploy the output content configuration to the user device 110 330, which proceeds to render the output content configuration until procurement / deployment 324 is complete. In this way, users can be provided with immersive content while waiting for resource deployment.

[0051] Fig. Figure 4A illustrates an exemplary flowchart for an exemplary process 400 for generating animated content using static images. It is understood that for this and other processes presented herein, additional, fewer, or alternative operations may be performed in a similar or alternative sequence, or at least partially in parallel, within the scope of protection of the various embodiments, unless otherwise specifically indicated. In this example, a first image corresponding to a foreground of an input image 402 is generated. The first image may include an image in which one or more foreground objects are extracted from the input image, as by using one or more machine learning systems for object identification, segmentation, and / or extraction. A second image may be generated corresponding to a first background of the input image 404.The first background can, for example, correspond to the original image without the extracted foreground. Alternatively, the first background can correspond to the original image in which an area designated as the foreground has been inpainted using one or more models.

[0052] In at least one embodiment, a third image can be generated corresponding to a second background of the input image 406. The second background can be associated with one or more outpainting processes, such as an image in which pixels of the background have been added compared to the original input image. An animated effect can be generated using the first image and at least one of the second image or the third image 408. For example, an animated effect can include zoom panning in one or all of the images used to create the effect, thereby creating the appearance of animation without the computational resources of video decoding.

[0053] Fig. Figure 4B illustrates an example flowchart of an example process 420 for providing a content configuration for rendering an animated effect on a user device. In this example, a request is received to access one or more resources associated with a distributed computing environment 422. For example, a user might request access to specific content or to use resources to run one or more applications, among other options. A wait time might be specified for acquiring and / or using the one or more resources, and it might be specified that the wait time exceeds a threshold 424.For example, a user may have an access "level" or tier associated with resources, and for some levels, the wait time may be long enough that it is desirable to provide the user with some immersive content to compensate for the wait and provide an enhanced experience. One or more content configurations can be selected from a content configuration database. The content configurations can be selected, at least in part, based on one or more properties of the request. For example, if a user has requested access to a specific video game, the one or more content configurations can be associated with that video game. As another example, if a user has a special status, an increased variety or complexity of content configurations may be available.The one or more content configurations can then be provided for rendering on a requesting device 482. In at least one embodiment, the content configurations can identify one or more images and an associated set of parameters for applying an animation effect. Accordingly, the user device can be used to render immersive content while the user waits for the resources to be deployed for use.

[0054] Fig. Figure 4C illustrates an exemplary flowchart for an exemplary process 440 for generating animated content using static images. In this example, a first image corresponding to the foreground of an input image 442 is generated. The first image may include an image in which one or more foreground objects are extracted from the input image, such as by using one or more machine learning systems for object identification, segmentation, and / or extraction. One or more second images may be generated corresponding to a background of the input image 444. The background may, for example, be the original image without the extracted foreground. As another example, the background may be the original image in which an area associated with the foreground has been inpainted using one or more models.As another example, the second image(s) may also be associated with an image created using one or more outpainting processes, such as an image in which pixels of the background have been added compared to the original input image. In at least one embodiment, an animated effect created using the first image and at least one of the second images may be displayed. 446 An animated effect may, for example, include zoom panning in one or all of the images used to create the effect, thereby creating the appearance of animation without the computational resource use of video decoding.

[0055] Fig. Figure 5 illustrates an exemplary flowchart of an exemplary process 500 for generating a content configuration for rendering an animated effect on a user device. In this example, one or more foreground components identified within an input image are extracted 502. The foreground components may be associated with one or more objects in the input image, which can be identified using one or more operations, object detection, segmentation, and / or combinations thereof. An initial image that includes at least one section containing the one or more foreground components 504 can be generated. Additionally, a mask region associated with the one or more foreground components 506 can be determined.For example, a mask can be created to provide a mask area in which one or more foreground components had a pixel value within the input image.

[0056] In at least one embodiment, one or more replacement pixels can be generated for the mask region 508. The one or more replacement pixels can be assigned to a generative AI system, such as a diffusion model, which can use an input prompt to generate the one or more replacement pixels. At least one second image, which includes the one or more replacement pixels in place of the mask region 510, can be generated. For example, a new background image can be generated that includes the background from the original input image but replaces the one or more foreground components with generated pixels. A configuration for performing an effect using the first image and a special second image of the at least one second image can be generated 512.The configuration can, for example, correspond to a JSON configuration that provides a desired effect, identification information for the first and second images, properties of the effect and / or the like.

[0057] As explained, aspects of various approaches presented herein may be simple enough to run in real time on a device such as a client device, like a personal computer or a game console. Such processing may be performed, for example, on content created or received on that client device or from an external source, such as streaming data or other content received over at least one network. In some cases, the processing and / or determination of this content may be performed by one of these other devices, systems, or entities, and then provided to the client device (or another such receiver) for presentation or other such use.

[0058] As an example, it illustrates Fig. 6 An exemplary network configuration 600 that can be used to provide, generate, modify, encode, process, and / or transmit image data or other such content. In at least one embodiment, a client device 602 can generate or receive data for a session using components of a control application 604 on the client device 602 and data stored locally on that client device. In at least one embodiment, a content application 624 running on a server 620 (e.g., a cloud server or an edge server) can initiate a session associated with at least one client device 602, such as a session manager might use, and user data stored in a user database 636, and can cause content, such as one or more digital assets (e.g.,Object representations) from an asset pool 634 are determined by a content manager 626. A content manager 626 can work with an image synthesis module 628 to create or synthesize new objects, digital assets, or other such content to be made available for presentation via the client device 602. In at least one embodiment, this image synthesis module 628 can use one or more neural networks or machine learning models that can be trained using a training module 632 or system located on or communicating with the server 620.This may include training and / or using a diffusion model 630 to generate content tiles that can be used by an image synthesis module 628, for example, to apply a non-repeating texture to an area of ​​an environment for which image or video data is to be presented via a client device 602. At least some of the generated content can be transmitted to the client device 602 using a suitable transmission manager 622 for sending by downloading, streaming, or another such transmission channel. An encoder can be used to encode and / or compress at least some of this data before transmission to the client device 602.In at least one embodiment, the client device 602, which receives such content, can provide this content to a corresponding control application 604, which may also or alternatively include a graphical user interface 610, a content manager 612, and an image synthesis or diffusion module 614 for use in providing, synthesizing, modifying, or using content for presentation (or other purposes) on or through the client device 602. A decoder can also be used to decode data received over the network(s) 640 for presentation via the client device 602, such as image or video content, through a display 606, and audio, such as sound and music, through at least one audio playback device 608, such as a loudspeaker or headphones.In at least one embodiment, at least some of this content can be stored, rendered, or made accessible to the client device 602, so that transmission over the network 640 is not required for at least some of the content, as if this content had been pre-downloaded or stored locally on a hard disk or optical disk. In at least one embodiment, a transmission mechanism, such as data streaming, can be used to transfer this content from the server 620 or the user database 636 to the client device 602. In at least one embodiment, at least some of this content can be received, enhanced, and / or streamed from another source, such as a third-party service 660 or another client device 650, which may also include a content application 662 for generating, enhancing, or providing content.In at least one embodiment, parts of this functionality can be performed using multiple computing devices or multiple processors within one or more computing devices that include a combination of CPUs and GPUs.

[0059] In this example, these client devices can include suitable computing devices such as a desktop computer, a notebook computer, a set-top box, a streaming device, a game console, a smartphone, a tablet computer, a VR headset, AR glasses, a portable computer, or a smart TV. Each client device can submit a request over at least one wired or wireless network, including, among others, the internet, Ethernet, a local area network (LAN), or a cellular network.In this example, these requests can be made to an address associated with a cloud provider, which can operate or control one or more electronic resources in the cloud provider's environment, such as a data center or server farm. In at least one embodiment, the request can be received or processed by at least one edge server located at a network edge and outside at least one security layer associated with the cloud provider's environment. This reduces latency by allowing client devices to interact with closer servers, while also improving the security of resources within the cloud provider's environment.

[0060] In at least one embodiment, such a system can be used to perform graphical rendering operations. In other embodiments, such a system can be used for other purposes, such as providing image or video content for testing or validating autonomous machine applications, or for performing deep learning operations. In at least one embodiment, such a system can be implemented using an edge device or can include one or more virtual machines (VMs). In at least one embodiment, such a system can be implemented at least partially in a data center or at least partially using cloud computing resources. INFERENCE AND TRAINING LOGIC

[0061] Fig. Figure 7A illustrates the inference and / or training logic 715, which is used to perform inference and / or training operations associated with one or more embodiments. Details of the inference and / or training logic 715 are given below in conjunction with the Fig. 7A and / or 7B provided.

[0062] In at least one embodiment, the inference and / or training logic 715 may, without limitation, include a code and / or data store 701 to store forward and / or output weights and / or input / output data and / or other parameters to configure neurons or layers of a neural network that is trained and / or used for inference in aspects of one or more embodiments.In at least one embodiment, the training logic 715 can include or be coupled to a code and / or data memory 701 for storing a graph code or other software for controlling the timing and / or order in which weighting and / or other parameter information is to be loaded to configure the logic, including integer and / or floating-point units (collectively referred to as ALUs (Arithmetic Logic Units)). In at least one embodiment, a code, such as a graph code, loads weighting or other parameter information into processor ALUs based on a neural network architecture to which the code corresponds.In at least one embodiment, the code and / or data memory 701 stores weighting parameters and / or input / output data of each layer of a neural network that is trained during the forward propagation of input / output data and / or weighting parameters during training and / or inference using aspects of one or more embodiments, or used in conjunction with one or more embodiments. In at least one embodiment, each part of the code and / or data memory 701 can be enclosed in another on-chip or off-chip data memory, including an L1, L2, or L3 cache or system memory of a processor.

[0063] In at least one embodiment, each part of the code and / or data memory 701 can be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data memory 701 can be a cache memory, a dynamic randomly addressable memory (“DRAM”), a static randomly addressable memory (“SRAM”), non-volatile memory (e.g., flash memory), or another type of memory. In at least one embodiment, the decision as to whether the code and / or data memory 701 is, for example, internal or external to a processor, or whether it consists of DRAM, SRAM, flash memory, or another type of memory, can depend on the on-chip or off-chip available memory, the latency requirements of training and / or inference functions performed, the batch size of the data used in inference and / or training of a neural network, or a combination of these factors.

[0064] In at least one embodiment, the inference and / or training logic 715 may, without limitation, comprise a code and / or data store 705 to store backward and / or output weighting and / or input / output data corresponding to neurons or layers of a neural network that is trained and / or used for inference in aspects of one or more embodiments. In at least one embodiment, the code and / or data store 705 stores weighting parameters and / or input / output data of each layer of a neural network that is trained or used in conjunction with one or more embodiments during the backward propagation of input / output data and / or weighting parameters during training and / or inference using aspects of one or more embodiments.In at least one embodiment, the training logic 715 can include or be coupled to a code and / or data memory 705 for storing a graph code or other software for controlling the timing and / or order in which weighting and / or other parameter information is to be loaded to configure the logic, including integer and / or floating-point units (collectively referred to as ALUs (arithmetic logic units)). In at least one embodiment, a code, such as a graph code, loads weighting or other parameter information into processor ALUs based on a neural network architecture to which the code corresponds. In at least one embodiment, each part of the code and / or data memory 705 can be enclosed in another on-chip or off-chip data memory, including an L1, L2, or L3 cache or system memory of a processor.In at least one embodiment, each part of the code and / or data storage 705 can be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data storage 705 can be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or another type of memory. In at least one embodiment, the decision as to whether the code and / or data storage 705 is, for example, internal or external to a processor, or whether it consists of DRAM, SRAM, flash memory, or another type of memory, can depend on the on-chip or off-chip available memory, the latency requirements of training and / or inference functions performed, the batch size of the data used in inference and / or training of a neural network, or a combination of these factors.

[0065] In at least one embodiment, the code and / or data memory 701 and the code and / or data memory 705 can be separate memory structures. In at least one embodiment, the code and / or data memory 701 and the code and / or data memory 705 can be the same memory structure. In at least one embodiment, the code and / or data memory 701 and the code and / or data memory 705 can be partly the same memory structure and partly different memory structures. In at least one embodiment, any part of the code and / or data memory 701 and the code and / or data memory 705 can be integrated into another on-chip or off-chip data memory, including the L1, L2, or L3 cache or system memory of a processor.

[0066] In at least one embodiment, the inference and / or training logic 715 can, without limitation, include one or more Arithmetic Logic Unit(s) (“ALU(s)”) 710, including integer and / or floating-point units, to perform logical and / or mathematical operations that are based at least partially on or specified by training and / or inference code (e.g., graph code), the result of which can generate activations (e.g., output values ​​of layers or neurons within a neural network) stored in an activation memory 720, which are functions of input / output and / or weighting parameter data stored in code and / or data memory 701 and / or code and / or data memory 705.In at least one embodiment, activations stored in the activation memory 720 are generated according to linear algebraic and / or matrix-based mathematics, which are executed by ALU(s) 710 in response to execution instructions or other code, wherein weight values ​​stored in the code and / or data memory 705 and / or code and / or data memory 701 are used as operands together with other values, such as bias values, gradient information, pulse values ​​or other parameters or hyperparameters, which may all or part be stored in the code and / or data memory 705 or code and / or data memory 701 or another memory on- or off-chip.

[0067] In at least one embodiment, the ALU(s) 710 are enclosed in one or more processors or other hardware logic devices or circuits, while in another embodiment, the ALU(s) 710 may be external to a processor or other hardware logic device or circuit that uses them (e.g., a co-processor). In at least one embodiment, the ALU(s) 710 may be enclosed in the execution units of a processor or otherwise in a bank of ALUs that the execution units of a processor can access, either within the same processor or distributed across different processors of different types (e.g., central processing units, graphics processing units, fixed functional units, etc.).In at least one embodiment, the code and / or data memory 701, the code and / or data memory 705, and the activation memory 720 may be located on the same processor or other hardware logic device or circuit, while in another embodiment they may be located on different processors or other hardware logic devices or circuits, or in a combination of the same and different processors or other hardware logic devices or circuits. In at least one embodiment, any part of the activation memory 720 may be integrated into another on-chip or off-chip data memory, including the L1, L2, or L3 cache or system memory of a processor.Furthermore, the inference and / or training code can be stored together with other code that a processor or other hardware logic or circuitry can access and that is retrieved and / or processed using the retrieval, decoding, scheduling, execution, elimination, and / or other logic circuitry of a processor.

[0068] In at least one embodiment, the activation memory 720 can be a cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or another type of memory. In at least one embodiment, the activation memory 720 can be located wholly or partially internally or externally to one or more processors or other logic circuits. In at least one embodiment, the choice of whether the activation memory 720 is internal or external to a processor, or whether it consists of DRAM, SRAM, flash, or another type of memory, can depend on the on-chip or off-chip available memory, the latency requirements of the training and / or inference functions being performed, the batch size of the data used in inference and / or training of a neural network, or a combination of these factors ... the activation memory 720 can depend on the latency requirements of the training and / or inference functions being performed. Fig. 7A illustrated inference and / or training logic 715 can be used in conjunction with an application-specific integrated circuit (“ASIC”), such as Google’s Tensorflow® Processing Unit, a Graphcore™ Inference Processing Unit (IPU), or an Intel Corp. Nervana® processor (e.g., “Lake Crest”). In at least one embodiment, the inference and / or training logic illustrated in Figure 715 can be used in conjunction with an application-specific integrated circuit (“ASIC”), such as Google’s Tensorflow® Processing Unit, a Graphcore™ Inference Processing Unit (IPU), or an Intel Corp. Nervana® Processor (e.g., “Lake Crest”). Fig. Figure 7A illustrates inference and / or training logic 715 in conjunction with central processing unit (CPU) hardware, graphics processing unit (GPU) hardware, or other hardware such as field-programmable gate arrays (FPGAs).

[0069] Fig. Figure 7B illustrates inference and / or training logic 715 according to at least one embodiment. In at least one embodiment, the inference and / or training logic 715 may, without limitation, include hardware logic in which computing resources are dedicated or otherwise used exclusively in conjunction with weight values ​​or other information corresponding to one or more layers of neurons within a neural network. In at least one embodiment, the Fig. Figure 7B illustrates inference and / or training logic 715 in conjunction with an application-specific integrated circuit (ASIC), such as Google's Tensorflow® Processing Unit, a Graphcore™ Inference Processing Unit (IPU), or an Intel Corp. Nervana® processor (e.g., "Lake Crest"). In at least one embodiment, the inference and / or training logic illustrated in Figure 7B can be used in conjunction with an application-specific integrated circuit (ASIC), such as Google's Tensorflow® Processing Unit, a Graphcore™ Inference Processing Unit (IPU), or an Intel Corp. Nervana® Processor (e.g., "Lake Crest"). In at least one embodiment, the logic illustrated in Figure 7B can be used in conjunction with an application-specific integrated circuit (ASIC), such as Google's Tensorflow® Processing Unit, a Graphcore™ Inference Processing Unit, or an Intel Corp. Nervana® Processor (e.g., "Lake Crest"). Fig. Figure 7B illustrates the use of inference and / or training logic 715 in conjunction with central processing unit (CPU) hardware, graphics processing unit (GPU) hardware, or other hardware, such as field-programmable gate arrays (FPGAs). In at least one embodiment, the inference and / or training logic 715 includes, without limitation, the code and / or data memory 701 and the code and / or data memory 705, which can be used to store code (e.g., graph code), weight values, and / or other information, including bias values, gradient information, pulse values, and / or other parameter or hyperparameter information. In at least one embodiment, Fig. In the embodiment illustrated in Figure 7B, each code and / or data store 701 and each code and / or data store 705 is connected to a dedicated computing resource, such as computing hardware 702 and computing hardware 706, respectively. In at least one embodiment, the computing hardware 702 and the computing hardware 706 each comprise one or more ALUs that perform mathematical functions, such as linear algebraic functions, only on information stored in the code and / or data store 701 and in the code and / or data store 705, respectively, with the result being stored in the activation memory 720.

[0070] In at least one embodiment, the code and / or data stores 701 and 705 and the corresponding computing hardware 702 and 706, respectively, correspond to different layers of a neural network, such that the resulting activation from a "memory / computing pair 701 / 702" consisting of code and / or data store 701 and computing hardware 702 is provided as an input for the "memory / computing pair 705 / 706" consisting of code and / or data store 705 and computing hardware 706, in order to reflect the conceptual organization of a neural network. In at least one embodiment, each of the memory / computing pairs 701 / 702 and 705 / 706 can correspond to more than one layer of the neural network. In at least one embodiment, additional memory / computing pairs (not shown) can be included in the inference and / or training logic 715 following or in parallel to the memory / computing pairs 701 / 702 and 705 / 706. DATA CENTER

[0071] Fig. Figure 8 illustrates an exemplary data center 800 in which at least one embodiment can be used. In at least one embodiment, the data center 800 includes a data center infrastructure layer 810, a framework layer 820, a software layer 830, and an application layer 840.

[0072] As in Fig. As shown in Figure 8, in at least one embodiment, the infrastructure layer 810 of the data center can include a resource orchestrator 812, clustered compute resources 814, and node compute resources (“node CRs”) 816(1)-816(N), where “N” is any positive integer. In at least one embodiment, the node CRs 816(1)-816(N) can include, but are not limited to, a number of central processing units (CPUs) or other processors (including DPUs, accelerators, field-programmable gate arrays (FPGAs), graphics processing units or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memories), storage devices (e.g., solid-state or disk drives), network input / output devices (NW I / O), network switches, virtual machines (VMs), power modules, and / or cooling modules, etc. In at least one embodiment, one or more node CRs of the node CR can be used.s 816(1)-816(N) be a server that has one or more of the computing resources mentioned above.

[0073] In at least one embodiment, grouped computing resources 814 can include separate groupings of node CRs housed in one or more racks (not shown), or many racks housed in data centers at different geographic locations (also not shown). Separate groupings of node CRs with grouped computing resources 814 can include grouped computing, networking, memory, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, multiple node CRs, including CPUs or processors, can be grouped in one or more racks to provide computing resources to support one or more workloads.In at least one embodiment, one or more racks can also include any number of power modules, cooling modules and / or network switches in any combination.

[0074] In at least one embodiment, the resource orchestrator 812 can configure or otherwise control one or more node CRs 816(1)-816(N) and / or grouped computing resources 814. In at least one embodiment, the resource orchestrator 812 can include a software design infrastructure management unit (“SDI” management unit) for the data center 800. In at least one embodiment, the resource orchestrator 812 can include hardware or software, or a combination thereof.

[0075] In at least one embodiment, as in Fig. As shown in Figure 8, the framework layer 820 includes a job scheduler 822, a configuration manager 824, a resource manager 826, and / or a distributed file system 828. In at least one embodiment, the framework layer 820 can include a framework for supporting software 832 of software layer 830 and / or one or more applications 842 of application layer 840. In at least one embodiment, the software 832 or the application(s) 842 can each include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. In at least one embodiment, the framework layer 820 can be a type of free and open-source software web application framework, such as Apache Spark™ (hereinafter referred to as “Spark”), which can utilize a distributed file system 828 for processing large amounts of data (e.g., “Big Data”) without being limited to it.In at least one embodiment, the job scheduler 822 can include a Spark driver to facilitate the scheduling of workloads supported by different layers of the data center 800. In at least one embodiment, the configuration manager 824 can be able to configure different layers, such as the software layer 830 and the framework layer 820, including Spark and the distributed file system 828 to support large-scale data processing. In at least one embodiment, the resource manager 826 can be able to manage clustered or grouped compute resources that are mapped to or associated with the distributed file system 828 and the job scheduler 822 for support. In at least one embodiment, the clustered or grouped compute resources can include the grouped compute resources 814 on the infrastructure layer 810 of the data center.In at least one embodiment, the resource manager 826 can coordinate with the resource orchestrator 812 to manage these mapped or allocated computing resources.

[0076] In at least one embodiment, the software 832 enclosed in the software layer 830 may include software used by at least sections of the node CRs 816(1)-816(N), grouped computing resources 814, and / or the distributed file system 828 of the framework layer 820. The one or more types of software may, but are not limited to, include internet web page search software, email virus scanning software, database software, and streaming video content software.

[0077] In at least one embodiment, the application(s) 842 enclosed in the application layer 840 may include one or more types of applications used by at least sections of the node CRs 816(1)-816(N), grouped compute resources 814, and / or the distributed file system 828 of the framework layer 820. One or more types of applications may, but are not limited to, include any number of genome applications, cognitive computation, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), or other machine learning applications used in conjunction with one or more embodiments.

[0078] In at least one embodiment, the configuration manager 824, the resource manager 826, and / or the resource orchestrator 812 can implement any number and any type of self-modifying actions based on any set and any type of data acquired in any technically feasible way. In at least one embodiment, self-modifying actions can relieve a data center operator of the data center 800 of potentially making poor configuration decisions and potentially avoiding underutilized and / or poorly functioning sections of a data center.

[0079] In at least one embodiment, the Data Center 800 may include tools, services, software, or other resources for training one or more machine learning models or for predicting or inferring information using one or more machine learning models according to one or more embodiments described herein. For example, in at least one embodiment, one or more machine learning models may be trained by calculating weighting parameters according to a neural network architecture, using software and / or computing resources described above with reference to the Data Center 800.In at least one embodiment, trained machine learning models corresponding to one or more neural networks can be used to derive or predict information using resources described above with reference to the Computing Center 800, using weighting parameters calculated by one or more training techniques such as those described herein.

[0080] In at least one embodiment, the data center can use CPUs, application-specific integrated circuits (ASICs), GPUs, FPGAs, and / or other hardware to perform training and / or inference using the resources described above. Furthermore, one or more of the software and / or hardware resources described above can be configured as a service to allow users to perform training or inference from information, such as image recognition, speech recognition, or other AI services.

[0081] The inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. Details of the inference and / or training logic 715 are given below in conjunction with the Fig. 7A and / or 7B are provided. In at least one embodiment, inference and / or training logic 715 can be provided in the system of Fig. 8 for inferring or predicting operations at least partly based on weighting parameters calculated using neural network training operations, neural network functions and / or architectures, or uses of the neural network described herein.

[0082] Such components can be used for content generation. COMPUTER SYSTEMS

[0083] Fig. Figure 9 is a block diagram illustrating an exemplary computer system, which may be a system with interconnected devices and components, a system-on-a-chip (SoC), or a combination thereof, 900, formed with a processor that may include execution units for carrying out an instruction according to at least one embodiment. In at least one embodiment, the computer system 900 may, without limitation, include a component, such as a processor 902, to employ execution units that carry out logic for executing algorithms on process data according to the present disclosure, as in the embodiment described herein.In at least one embodiment, the Computer System 900 may include processors such as the PENTIUM® processor family, Xeon™, Itanium®, XScale™ and / or StrongARM™, Intel® Core™ or Intel® Nervana™ microprocessors available from Intel Corporation in Santa Clara, California, although other systems (including PCs incorporating other microprocessors, engineering workstations, set-top boxes, and the like) may be used. In at least one embodiment, the Computer System 900 may run a version of the WINDOWS operating system available from Microsoft Corporation, Redmond, Washington, although other operating systems (for example, UNIX and Linux), embedded software, and / or graphical user interfaces may be used.

[0084] Embodiments can also be used in other devices, such as handheld devices and embedded applications. Some examples of handheld devices include mobile phones, Internet Protocol devices, digital cameras, personal digital assistants (“PDAs”), and handheld PCs. In at least one embodiment, embedded applications can include a microcontroller, a digital signal processor (“DSP”), a system-on-a-chip, network computers (“NetPCs”), set-top boxes, network hubs, wide-area network switches (“WAN” switches), or any other system capable of executing one or more instructions according to at least one embodiment.

[0085] In at least one embodiment, the computer system 900 can, without limitation, include a processor 902, which can, without limitation, include one or more execution units 908 for training and / or inferring a machine learning model according to the techniques described herein. In at least one embodiment, the computer system 900 is a single-processor desktop or server system, but in another embodiment, the computer system 900 can be a multiprocessor system.In at least one embodiment, the processor 902 can, without limitation, include a complex instruction set computing microprocessor (“CISC” microprocessor), a reduced instruction set computing microprocessor (“RISC” microprocessor), a very long instruction word computing microprocessor (“VLIW” microprocessor), a processor implementing a combination of instruction sets, or any other processing device, such as a digital signal processor. In at least one embodiment, the processor 902 can be coupled to a processor bus 910, which can transmit data signals between the processor 902 and other components in the computer system 900.

[0086] In at least one embodiment, the processor 902 can, without limitation, include an internal Level 1 ("L1") cache memory ("cache") 904. In at least one embodiment, the processor 902 can have a single internal cache or multiple levels of an internal cache. In at least one embodiment, the cache memory can be located external to the processor 902. Other embodiments can also include a combination of internal and external caches, depending on the specific implementation and requirements. In at least one embodiment, the register file 906 can store different data types in different registers, including, without limitation, integer registers, floating-point registers, status registers, and instruction pointer registers.

[0087] In at least one embodiment, the execution unit 908, including, without limitation, logic for performing integer and floating-point operations, is also located in the processor 902. In at least one embodiment, the processor 902 may also include a read-only memory (“ROM”) for microcode (“ucode”) that stores microcode for specific macro instructions. In at least one embodiment, the execution unit 908 may include logic for handling a packed instruction set 909. In at least one embodiment, by including the packed instruction set 909 in an instruction set of a general-purpose processor 902, together with associated circuitry for executing instructions, operations used by many multimedia applications can be performed using packed data in a general-purpose processor 902.In one or more embodiments, many multimedia applications can be accelerated and run more efficiently by utilizing the full width of a processor's data bus to perform operations on packed data, thereby eliminating the need to transfer smaller data units across the processor's data bus to perform one or more operations on each data element.

[0088] In at least one embodiment, the execution unit 908 can also be used in microcontrollers, embedded processors, graphics devices, DSPs, and other logic circuit types. In at least one embodiment, the computer system 900 can, without limitation, include a main memory 920. In at least one embodiment, the memory 920 can be implemented as a dynamic random-access memory device (“DRAM” device), a static random-access memory device (“SRAM” device), a flash memory device, or another main memory device. In at least one embodiment, the main memory 920 can store instructions 919 and / or data 921, represented by data signals that can be executed by the processor 902.

[0089] In at least one embodiment, the system logic chip can be coupled to the processor bus 910 and the main memory 920. In at least one embodiment, the system logic chip can, without restriction, include a memory controller hub (“MCH”) 916, and the processor 902 can communicate with the MCH 916 via the processor bus 910. In at least one embodiment, the MCH 916 can provide a high-bandwidth main memory path 918 to the main memory 920 for storing instructions and data and for storing graphics instructions, data, and textures. In at least one embodiment, the MCH 916 can route data signals between the processor 902, the main memory 920, and other components in the computer system 900 and bridge data signals between the processor bus 910, the main memory 920, and a system I / O 922.In at least one embodiment, the system logic chip can provide a graphics port for coupling with a graphics controller. In at least one embodiment, the MCH 916 can be coupled to the main memory 920 via a high-bandwidth memory path 918, and the graphics / video card 912 can be coupled to the MCH 916 via an Accelerated Graphics Port (“AGP” connection) 914.

[0090] In at least one embodiment, the computer system 900 can use a system I / O 922, which is a proprietary hub interface bus, to couple the MCH 916 to the I / O controller hub (“ICH”) 930. In at least one embodiment, the ICH 930 can provide direct connections to some I / O devices via a local I / O bus. In at least one embodiment, the local I / O bus can, without limitation, include a high-speed I / O bus for connecting peripheral devices to the main memory 920, the chipset, and the processor 902. Examples of such peripheral devices can, without limitation, include an audio controller 929, a firmware hub (“flash BIOS”) 928, a wireless transceiver 926, a data storage device 924, a legacy I / O controller 923 that includes user input and keyboard interfaces 925, a serial expansion port 927, etc. B. Universal Serial Bus (“USB”), and include a 934 network controller.The 924 data storage device can include a hard disk drive, a floppy disk drive, a CD-ROM device, a flash memory device, or any other mass storage device.

[0091] Illustrated in at least one embodiment Fig. 9 a system that includes interconnected hardware devices or “chips”, while Fig. 9 in other embodiments can illustrate an exemplary system on a chip (“SoC”). In at least one embodiment, devices can be interconnected via proprietary connections, standardized connections (e.g., PCIe), or a combination thereof. In at least one embodiment, one or more components of the computer system 900 are interconnected via Compute Express Link links (CXL links).

[0092] The inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. Details of the inference and / or training logic 715 are given below in conjunction with the Fig. 7A and / or 7B are provided. In at least one embodiment, inference and / or training logic 715 can be provided in the system of Fig. 9 for inferring or predicting operations at least partly based on weighting parameters calculated using neural network training operations, neural network functions and / or architectures, or uses of the neural network described herein.

[0093] Such components can be used for content generation.

[0094] Fig. Figure 10 is a block diagram illustrating an electronic device 1000 for using a processor 1010 according to at least one embodiment. In at least one embodiment, the electronic device 1000 can be, for example, and without limitation, a notebook, a tower server, a rack server, a blade server, a laptop, a desktop computer, a tablet, a mobile device, a telephone, an embedded computer, or another suitable electronic device.

[0095] In at least one embodiment, the electronic device 1000 can include, without limitation, a processor 1010 that is communicatively coupled to any number or type of components, peripherals, modules, or devices. In at least one embodiment, the processor 1010 is coupled using a bus or interface, such as a 1°C bus, a system management bus (“SMBus”), a low-pin-count bus (“LPC” bus), a serial peripheral interface (“SPI”), a high-definition audio bus (“HDA” bus), a serial advance technology attachment bus (“SATA” bus), a universal serial bus (“USB”) (versions 1, 2, 3), or a universal asynchronous receiver / transmitter bus (“UART” bus). In at least one embodiment, the following is illustrated: Fig. 10 a system that includes interconnected hardware devices or “chips”, while Fig. 10. In other embodiments, an exemplary system-on-chip (“SoC”) can be illustrated. In at least one embodiment, the Fig. The devices illustrated in 10 may be connected using proprietary connections, standardized connections (e.g., PCIe), or a combination thereof. In at least one embodiment, one or more components of the Fig. 10 are interconnected via Compute Express Link links (“CXL” links).

[0096] In at least one embodiment, Fig. 10 a display 1024, a touchscreen 1025, a touchpad 1030, a near-field communications unit (“NFC”) 1045, a sensor hub 1040, a thermal sensor 1046, an Express Chipset (“EC”) 1035, a Trusted Platform Module (“TPM”) 1038, BIOS / firmware / flash memory (“BIOS, FW flash memory”) 1022, a DSP 1060, a drive 1020, such as a solid-state drive (“SSD”) or a hard disk drive (“HDD”), a wireless local area network (“WLAN” unit) 1050, a Bluetooth unit 1052, a wireless wide area network (“WWAN” unit) 1056, a global positioning system (GPS) 1055, a camera (“USB 3.0 camera”) 1054, such as a USB 3.0 camera, and / or a low-power double data rate memory unit (“LPDDR3” memory unit) 1015, which is implemented, for example, in the LPDDR3 standard. These components can be implemented in any suitable manner.

[0097] In at least one embodiment, other components can be communicatively coupled to the processor 1010 via the components described above. In at least one embodiment, an accelerometer 1041, an ambient light sensor (“ALS”) 1042, a compass 1043, and a gyroscope 1044 can be communicatively coupled to the sensor hub 1040. In at least one embodiment, the thermal sensor 1039, a fan 1037, a keyboard 1036, and a touchpad 1030 can be communicatively coupled to the EC 1035. In at least one embodiment, loudspeakers 1063, headphones 1064, and a microphone (“mic”) 1065 can be communicatively coupled to an audio unit (“Audio Codec and Class d Amp”) 1062, which in turn can be communicatively coupled to the 1060. In at least one embodiment, the audio unit 1062 can, for example and without limitation, include an audio encoder / decoder (“codec”) and a class D amplifier.In at least one embodiment, the SIM card (“SIM”) 1057 can be communicatively coupled with the WWAN unit 1056. In at least one embodiment, components such as the WLAN unit 1050 and the Bluetooth unit 1052, as well as the WWAN unit 1056, can be implemented in a Next Generation Form Factor (“NGFF”).

[0098] The inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. Details of the inference and / or training logic 715 are given below in conjunction with the Fig. 7A and / or 7B are provided. In at least one embodiment, inference and / or training logic 715 can be provided in the system of Fig. 10 for inferring or predicting operations at least partly based on weighting parameters calculated using neural network training operations, neural network functions and / or architectures or uses of the neural network described herein.

[0099] Such components can be used for content generation.

[0100] Fig. Figure 11 is a block diagram of a processing system according to at least one embodiment. In at least one embodiment, the system 1100 includes one or more processors 1102 and one or more graphics processors 1108, and can be a single-processor desktop system, a multi-processor workstation system, or a server system with a large number of processors 1102 or processor cores 1107. In at least one embodiment, the system 1100 is a processing platform integrated into an integrated system-on-a-chip (SoC) circuit for use in mobile, portable, or embedded devices.

[0101] In at least one embodiment, the system 1100 can include or be integrated with a server-based gaming platform, a gaming console (including a gaming and media console), a mobile gaming console, a handheld gaming console, or an online gaming console. In at least one embodiment, the system 1100 is a mobile phone, a smartphone, a tablet computer, or a mobile internet device. In at least one embodiment, the processing system 1100 can also include, be coupled to, or be integrated with a wearable device, such as a wearable smartwatch, smart glasses, an augmented reality device, or a virtual reality device.In at least one embodiment, the processing system 1100 is a television or set-top box device comprising one or more processors 1102 and a graphical interface generated by one or more graphics processors 1108.

[0102] In at least one embodiment, one or more processors 1102 each include one or more processor cores 1107 for processing instructions that, when executed, perform operations for system and application software. In at least one embodiment, one or more processor cores 1107 are each configured to process a specific instruction set 1109. In at least one embodiment, the instruction set 1109 can facilitate Complex Instruction Set Computing (CISC), Reduced Instruction Set Computing (RISC), or computation using a Very Long Instruction Word (VLIW). In at least one embodiment, the processor cores 1107 can each process a different instruction set 1109, which may include instructions for facilitating the emulation of other instruction sets. In at least one embodiment, the processor core 1107 can also include other processing devices, such as a digital signal processor (DSP).

[0103] In at least one embodiment, the processor 1102 includes a cache memory 1104. In at least one embodiment, the processor 1102 can have a single internal cache or multiple levels of internal cache. In at least one embodiment, the cache memory is shared by different components of the processor 1102. In at least one embodiment, the processor 1102 also uses an external cache (e.g., a Level 3 cache (L3 cache) or Last Level Cache (LLC)) (not shown), which can be shared by the processor core(s) 1107 using known cache coherence techniques. In at least one embodiment, the register file 1106 is additionally included in the processor 1102, which can include different types of registers for storing different types of data (e.g., integer registers, floating-point registers, status registers, and an instruction pointer register).In at least one embodiment, the register file can include 1106 universal registers or other registers.

[0104] In at least one embodiment, one or more processors 1102 are coupled to one or more interface buses 1110 to transmit communication signals, such as address, data, or control signals, between the processor(s) 1102 and other components in the system 1100. In at least one embodiment, the interface bus 1110 can be a processor bus, such as a version of a Direct Media Interface (DMI) bus. In at least one embodiment, the interface bus(s) 1110 is / are not limited to a DMI bus but can include one or more peripheral component interconnect buses (e.g., PCI, PCI Express), memory buses, or other interface bus types. In at least one embodiment, the processor(s) 1102 includes an integrated memory controller 1116 and a platform controller hub 1130.In at least one embodiment, the main memory controller 1116 facilitates communication between a main memory device and other components of the system 1100, while the platform controller hub (PCH) 1130 provides connections to I / O devices via a local I / O bus.

[0105] In at least one embodiment, the memory device 1120 can be a dynamic random-access memory (DRAM) device, a static random-access memory (SRAM) device, a flash memory device, a phase-change memory device, or another memory device that has suitable performance to serve as process memory. In at least one embodiment, the memory device 1120 can operate as system memory for the system 1100 to store data 1122 and instructions 1121 for use when one or more processors 1102 execute an application or process. In at least one embodiment, the memory controller 1116 is also coupled with an optional external graphics processor 1112, which can communicate with one or more graphics processors 1108 in processors 1102 to perform graphics and media operations.In at least one embodiment, a display device 1111 can be connected to one or more processors 1102. In at least one embodiment, the display device 1111 can include an internal display device, such as in a mobile electronic device or a laptop device, and / or an external display device connected via a display interface (e.g., DisplayPort, etc.). In at least one embodiment, the display device 1111 can include a head-mounted display (HDM), such as a stereoscopic display device for use in virtual reality (VR) or augmented reality (AR) applications.

[0106] In at least one embodiment, the platform controller hub 1130 enables the connection of peripheral devices to the memory device 1120 and the processor(s) 1102 via a high-speed I / O bus. In at least one embodiment, the I / O peripheral devices include, but are not limited to, an audio controller 1146, a network controller 1134, a firmware interface 1128, a wireless transceiver 1126, touch sensors 1125, and a data storage device 1124 (e.g., a hard disk drive, flash memory, etc.). In at least one embodiment, the data storage device 1124 can be connected via a storage interface (e.g., SATA) or via a peripheral bus, such as a peripheral component interconnect bus (e.g., PCI, PCI Express). In at least one embodiment, the touch sensors can include 1125 touchscreen sensors, pressure sensors or fingerprint sensors.In at least one embodiment, the wireless transceiver 1126 can be a WiFi transceiver, a Bluetooth transceiver, or a cellular network transceiver, such as a 3G, 4G, or Long-Term Evolution (LTE) transceiver. In at least one embodiment, the firmware interface 1128 enables communication with system firmware and can, for example, be a Unified Extensible Firmware Interface (UEFT). In at least one embodiment, the network controller 1134 can enable network connectivity to a wired network. In at least one embodiment, a high-performance network controller (not shown) is coupled to the interface bus(s) 1110. In at least one embodiment, the audio controller 1146 is a multi-channel high-definition audio controller. In at least one embodiment, the system 1100 includes an optional legacy I / O controller 1140 for coupling legacy devices (e.g.,Personal System 2 (PS / 2) devices are integrated with the system. In at least one embodiment, the platform controller hub 1130 can also be connected to one or more Universal Serial Bus controllers (USB controller(s)) 1142 with input devices, such as keyboard and mouse combinations 1143, a camera 1144, or other USB input devices.

[0107] In at least one embodiment, an instance of the memory controller 1116 and the platform controller hub 1130 can be integrated into a discrete external graphics processor, such as the external graphics processor 1112. In at least one embodiment, the platform controller hub 1130 and / or the memory controller 1116 can be external to one or more processors 1102. For example, in at least one embodiment, the system 1100 can include an external memory controller 1116 and a platform controller hub 1130, which can be configured as a memory controller hub and peripheral controller hub within a system chipset that communicates with the processor(s) 1102.

[0108] The inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. Details of the inference and / or training logic 715 are given below in conjunction with the Fig. 7A and / or 7B provided. In at least one embodiment, the inference and / or training logic 715 can be wholly or partially integrated into the graphics processor 1500. For example, in at least one embodiment, training and / or inference techniques described herein can use one or more ALUs embodied in a graphics processor. Moreover, in at least one embodiment, inference and / or training operations described herein can be performed using logic other than that provided in the Fig. The logic illustrated in Figures 7A and / or 7B can be performed. In at least one embodiment, weighting parameters can be stored in on-chip or off-chip memory and / or registers (shown or not shown) that configure graphics processor ALUs to perform one or more machine learning algorithms, neural network architectures, use cases, or training techniques described herein.

[0109] Such components can be used for content generation.

[0110] Fig. Figure 12 is a block diagram of a processor 1200 comprising one or more processor cores 1202A-1202N, an integrated memory controller 1214, and an integrated graphics processor 1208 according to at least one embodiment. In at least one embodiment, the processor 1200 may include additional cores up to and including the additional core 1202N, which is represented by dashed boxes. In at least one embodiment, each of the processor cores 1202A-1202N includes one or more internal cache units 1204A-1204N. In at least one embodiment, each processor core also has access to one or more shared cache units 1206.

[0111] In at least one embodiment, the internal cache unit(s) 1204A-1204N and the shared cache units 1206 constitute a cache memory hierarchy in the processor 1200. In at least one embodiment, the cache units 1204A-1204N can include at least one level of instruction and data cache within each processor core and one or more levels of shared mid-level cache, such as Level 2 (L2), Level 3 (L3), Level 4 (L4), or other cache levels, the highest cache level prior to external memory being classified as an LLC. In at least one embodiment, the cache coherence logic maintains coherence between different cache units 1206 and 1204A-1204N.

[0112] In at least one embodiment, the processor 1200 can also include a set of one or more bus controller units 1216 and a system agent core 1210. In at least one embodiment, one or more bus controller units 1216 manage a number of peripheral buses, such as one or more PCI or PCI Express buses. In at least one embodiment, the system agent core 1210 provides management functions for various processor components. In at least one embodiment, the system agent core 1210 includes one or more integrated memory controllers 1214 for managing access to various external memory devices (not shown).

[0113] In at least one embodiment, one or more processor core(s) 1202A-1202N include support for simultaneous multithreading. In at least one embodiment, the system agent core 1210 includes components for coordinating processor cores 1202A-1202N during multithreaded processing. In at least one embodiment, the system agent core 1210 may additionally include a power control unit (PCU) that includes logic and components for regulating one or more power supply states of the processor core(s) 1202A-1202N and the graphics processor 1208.

[0114] In at least one embodiment, the processor 1200 additionally includes the graphics processor 1208 for performing graphics processing operations. In at least one embodiment, the graphics processor 1208 is coupled with shared cache unit(s) 1206 and the system agent core 1210, including one or more integrated memory controllers 1214. In at least one embodiment, the system agent core 1210 also includes a display controller 1211 for driving the graphics processor output to one or more coupled displays. In at least one embodiment, the display controller 1211 can also be a separate module coupled to the graphics processor 1208 via at least one connection, or it can be integrated into the graphics processor 1208.

[0115] In at least one embodiment, a ring-based interconnect 1212 is used to couple internal components of the processor 1200. In at least one embodiment, an alternative interconnect, such as a point-to-point connection, a switched connection, or other techniques, can be used. In at least one embodiment, the graphics processor 1208 is coupled to a ring-based interconnect 1212 via an I / O link 1213.

[0116] In at least one embodiment, the I / O link 1213 represents at least one of several I / O link variants, including a package I / O link, which facilitates communication between different processor components and an embedded high-performance memory module 1218, for example, an eDRAM module. In at least one embodiment, each processor core(s) 1202A-1202N and the graphics processor 1208 use embedded memory modules 1218 as a shared last-level cache.

[0117] In at least one embodiment, the processor cores 1202A-1202N are homogeneous cores executing a common instruction set architecture. In at least one embodiment, the processor cores 1202A-1202N are heterogeneous with respect to the instruction set architecture (ISA), wherein one or more processor cores 1202A-1202N execute a common instruction set, while one or more other processor cores 1202A-1202N execute a subset of a common instruction set or a different instruction set. In at least one embodiment, the processor core(s) 1202A-1202N are heterogeneous with respect to the microarchitecture, wherein one or more cores with a relatively higher power consumption are coupled with one or more power cores with a lower power consumption. In at least one embodiment, the processor 1200 can be implemented on one or more chips or as an integrated SoC circuit.

[0118] The inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. Details of the inference and / or training logic 715 are given below in conjunction with the Fig. 7A and / or 7B are provided. In at least one embodiment, the inference and / or training logic 715 can be wholly or partially integrated into the processor 1200. For example, in at least one embodiment, the training and / or inference techniques described herein can use one or more of the ALUs located in the graphics processor 1208, in the graphics cores 1202A-1202N, or in other components in Fig. 12 are embodied. Moreover, in at least one embodiment, the inference and / or training operations described herein can be performed using a logic other than that described in the Fig. The logic illustrated in Figures 7A and / or 7B can be performed. In at least one embodiment, weighting parameters can be stored in on-chip or off-chip memory and / or registers (shown or not shown) that configure the ALUs of the 1200 graphics processor to perform one or more of the machine learning algorithms, neural network architectures, use cases, or training techniques described herein.

[0119] Such components can be used for content generation. VIRTUALIZED COMPUTING PLATFORM

[0120] Fig. Figure 13 is an exemplary data flow diagram for a process 1300 for generating and deploying an image processing and inference pipeline according to at least one embodiment. In at least one embodiment, the process 1300 can be used with imaging devices, processing devices, and / or other types of devices in one or more facilities 1302. The process 1300 can be executed within a training system 1304 and / or a deployment system 1306. In at least one embodiment, the training system 1304 can be used to perform training, deployment, and implementation of machine learning models (e.g., neural networks, object detection algorithms, computer vision algorithms, etc.) for use in the deployment system 1306.In at least one embodiment, the deployment system 1306 can be configured to offload processing and computing resources to a distributed computing environment in order to reduce infrastructure requirements in the facility 1302. In at least one embodiment, one or more applications in a pipeline can utilize or call services (e.g., inference, visualization, computing, AI, etc.) of the deployment system 1306 when executing applications.

[0121] In at least one embodiment, some of the applications used in advanced processing and inference pipelines can use machine learning models or other AI to perform one or more processing steps. In at least one embodiment, machine learning models in the facility 1302 can be trained using data 1308 (e.g., imaging data) generated in the facility 1302 (and stored on one or more Picture Archiving and Communication System (PACS) servers in the facility 1302), or trained using imaging or sequencing data 1308 from one or more other facilities, or a combination thereof. In at least one embodiment, the training system 1304 can be used to provide applications, services, and / or other resources for generating functional, deployable machine learning models for the deployment system 1306.

[0122] In at least one embodiment, the model register 1324 can be supported by an object store that can support versioning and object metadata. In at least one embodiment, the object store can be accessed, for example, via an application programming interface (API) compatible with cloud storage within a cloud platform. In at least one embodiment, machine learning models in the model register 1324 can be uploaded, listed, modified, or deleted by developers or partners of a system interacting with an API. In at least one embodiment, an API can provide access to procedures that allow users with appropriate credentials to associate models with applications, so that models can be executed as part of the execution of containerized instantiations of applications.

[0123] In at least one embodiment, the training system 1304 ( Fig. 13) include a scenario in which the facility 1302 trains its own machine learning model or has an existing machine learning model that needs to be optimized or updated. In at least one embodiment, imaging data 1308 generated by imaging devices, sequencing devices, and / or other types of devices can be received. In at least one embodiment, after receiving the image data 1308, AI-assisted annotation 1310 can be used to assist in generating annotations that correspond to image data 1308 used as basic truth data for a machine learning model. In at least one embodiment, AI-assisted annotation 1310 can include one or more machine learning models (e.g., convolutional neural networks, CNNs) that can be trained to generate annotations that correspond to certain types of image data 1308 (e.g., from certain devices).In at least one embodiment, AI-supported annotations 1310 can then be used directly or adjusted or fine-tuned with an annotation tool to generate basic truth data. In at least one embodiment, AI-supported annotations 1310, labeled data 1312, or a combination thereof can be used as basic truth data for training a machine learning model. In at least one embodiment, a trained machine learning model can be designated as output model(s) 1316 and used by the deployment system 1306 as described herein.

[0124] In at least one embodiment, a training pipeline can include a scenario in which the facility 1302 requires a machine learning model for use in performing one or more processing tasks for one or more applications in the deployment system 1306, but the facility 1302 may not currently have such a machine learning model (or may not have a model that is optimized, efficient, or effective for such purposes). In at least one embodiment, an existing machine learning model can be selected from a model register 1324. In at least one embodiment, the model register 1324 can include machine learning models that have been trained to perform a variety of different inference tasks on imaging data.In at least one embodiment, machine learning models in model register 1324 may have been trained on imaging data from facilities other than facility 1302 (e.g., facilities at a different location). In at least one embodiment, machine learning models may have been trained on imaging data from one location, two locations, or any number of locations. In at least one embodiment, if training is performed on imaging data from a specific location, it may take place at that location or at least in a manner that protects the confidentiality of the imaging data or restricts the transfer of imaging data outside of that location. In at least one embodiment, once a machine learning model has been trained—or partially trained—at a location, it may be added to model register 1324.In at least one embodiment, a machine learning model can then be retrained or updated at any number of other facilities, and a retrained or updated model can be made available in the model register 1324. In at least one embodiment, a machine learning model can then be selected from the model register 1324—and designated as output model(s) 1316—and used in the deployment system 1306 to perform one or more processing tasks for one or more applications of a deployment system.

[0125] In at least one embodiment, a scenario may include the facility 1302, which requires a machine learning model for use in performing one or more processing tasks for one or more applications in the deployment system 1306. The facility 1302 may not currently have such a machine learning model (or may not have a model that is optimized, efficient, or effective for such purposes). In at least one embodiment, a machine learning model selected from the model register 1324 may not be fine-tuned or optimized for imaging data 1308 generated in the facility 1302 due to differences in populations, robustness of the training data used to train a machine learning model, diversity of anomalies in the training data, and / or other problems with the training data.In at least one embodiment, AI-assisted annotation 1310 can be used to assist in generating annotations corresponding to imaging data 1308, which are used as baseline truth data for retraining or updating a machine learning model. In at least one embodiment, labeled data 1312 can be used as baseline truth data for training a machine learning model. In at least one embodiment, retraining or updating a machine learning model can be referred to as model training 1314. In at least one embodiment, model training 1314—e.g., AI-assisted annotations 1310, labeled data 1312, or a combination thereof—can be used as baseline truth data for retraining or updating a machine learning model.In at least one embodiment, a trained machine learning model can be designated as output model(s) 1316 and used by the deployment system 1306 as described herein.

[0126] In at least one embodiment, the deployment system 1306 can include software 1318, services 1320, hardware 1322, and / or other components, features, and functions. In at least one embodiment, the deployment system 1306 can include a software "stack" such that software 1318 can be built on top of services 1320 and use services 1320 to perform some or all of its processing tasks, and services 1320 and software 1318 can be built on top of hardware 1322 and use hardware 1322 to perform processing, storage, and / or other computational tasks of the deployment system 1306. In at least one embodiment, the software 1318 can contain any number of different containers, each container capable of executing an instantiation of an application.In at least one embodiment, each application can perform one or more processing tasks in an advanced processing and inference pipeline (e.g., inference, object detection, feature recognition, segmentation, image enhancement, calibration, etc.). In at least one embodiment, an advanced processing and inference pipeline can be defined based on the selection of various containers that are desired or required for processing imaging data 1308, in addition to containers that receive and configure imaging data for use by the individual containers and / or for use by the device 1302 after processing by a pipeline (e.g., for converting outputs back into a usable data type). In at least one embodiment, a combination of containers within the software 1318 (e.g.,a pipeline) is referred to as a virtual instrument (as described in more detail herein), and a virtual instrument can use services 1320 and hardware 1322 to perform some or all of the processing tasks of applications instantiated in containers.

[0127] In at least one embodiment, a data processing pipeline can receive input data (e.g., imaging data 1308) in a specific format in response to an inference request (e.g., a request from a user of the deployment system 1306). In at least one embodiment, input data can represent one or more images, videos, and / or other data representations generated by one or more imaging devices. In at least one embodiment, data can be preprocessed as part of the data processing pipeline to prepare it for processing by one or more applications.In at least one embodiment, post-processing can be performed on the output of one or more inference tasks or other processing tasks of a pipeline to prepare output data for a subsequent application and / or to prepare output data for transmission and / or use by a user (e.g., in response to an inference request). In at least one embodiment, inference tasks can be performed by one or more machine learning models, such as trained or deployed neural networks, which may include output models 1316 of the training system 1304.

[0128] In at least one embodiment, tasks of the data processing pipeline can be encapsulated in one or more containers, each representing a discrete, fully functional instantiation of an application and a virtualized computing environment capable of referencing machine learning models. In at least one embodiment, containers or applications can be published in a private (e.g., access-restricted) area of ​​a container register (described in more detail herein), and trained or deployed models can be stored in the model register 1324 and assigned to one or more applications. In at least one embodiment, images from applications (e.g.,Container images) may be available in a container register, and after selection by a user from a container register for use in a pipeline, an image can be used to create a container for instantiating an application for use by a user's system.

[0129] In at least one embodiment, developers (e.g., software developers, clinicians, physicians, etc.) can develop, publish, and store applications (e.g., as containers) to perform image processing and / or inference on supplied data. In at least one embodiment, development, publication, and / or storage can be performed using a system-associated Software Development Kit (SDK) (e.g., to ensure that a developed application and / or container is compliant or compatible with a system). In at least one embodiment, a developed application can be tested locally (e.g., in a first facility, on data from a first facility) using an SDK that can support at least some of the services described in section 1320 as a system (e.g., the system described in section 1200). Fig. 12) Since DICOM objects can contain between one and hundreds of images or other data types, and due to data variation, in at least one embodiment a developer may be responsible for managing (e.g., defining constructs, incorporating preprocessing into an application, etc.) the extraction and preparation of incoming data. In at least one embodiment, once validated by System 1300 (e.g., for accuracy), an application may be available in a container register for selection and / or implementation by a user to perform one or more processing tasks with respect to data in a user's facility (e.g., a second facility).

[0130] In at least one embodiment, developers can then make applications or containers available over a network for access and use by users of a system (e.g., the System 1300 of the Fig. 13) share. In at least one embodiment, completed and validated applications or containers can be stored in a container register, and associated machine learning models can be stored in the model register 1324. In at least one embodiment, a requesting entity—providing an inference or image processing request—can search a container register and / or model register 1324 for an application, container, dataset, machine learning model, etc., select a desired combination of elements to include in the data processing pipeline, and submit it to an image processing request. In at least one embodiment, a request can include input data required to execute a request and / or can include a selection of applications and / or machine learning models to be executed when processing a request.In at least one embodiment, a request can then be forwarded to one or more components of the deployment system 1306 (e.g., a cloud) to perform the processing of the data processing pipeline. In at least one embodiment, the processing by the deployment system 1306 can include referencing selected elements (e.g., applications, containers, models, etc.) from a container register and / or model register 1324. In at least one embodiment, the results generated by a pipeline can be returned to a user as a reference (e.g., for viewing in a viewing application suite running on a local workstation or a local terminal).

[0131] In at least one embodiment, services 1320 can be used to support the processing or execution of applications or containers in pipelines. In at least one embodiment, the services 1320 can include computing services, AI services (artificial intelligence services), visualization services, and / or other service types. In at least one embodiment, the services 1320 can provide functions common to one or more applications in the software 1318, so that the functions can be abstracted into a service that can be called or used by applications. In at least one embodiment, the functionality provided by services 1320 can run dynamically and more efficiently while scaling well by allowing applications to process data in parallel (e.g., using a parallel computing platform 1230). Fig. 12)). In at least one embodiment, services 1320 can be shared between and among different applications, instead of each application needing its own instance of services 1320 to perform the same functionality provided by services 1320. In at least one embodiment, services can include an inference server or engine that can be used to perform collection or segmentation tasks as non-restrictive examples. In at least one embodiment, a model training service can be included that can provide training and / or retraining capabilities for machine learning models. In at least one embodiment, a data augmentation service can also be included that can extract, resize, scale, and / or otherwise augment GPU-accelerated data (e.g., DICOM, RIS, CIS, REST-compliant data, RPC, raw data, etc.).In at least one embodiment, a visualization service can be used that can add image rendering effects—such as ray tracing, rasterization, noise reduction, sharpening, etc.—to enhance the realism of two-dimensional (2D) and / or three-dimensional (3D) models. At least one embodiment can also include virtual instrument services that provide beamforming, segmentation, inference, imaging, and / or support for other applications within virtual instrument pipelines.

[0132] In at least one embodiment, where Services 1320 include an AI service (e.g., an inference service), one or more machine learning models can be executed by invoking an inference service (e.g., an inference server) (e.g., as an API call) to execute one or more machine learning models or their processing as part of application execution. In at least one embodiment, where another application includes one or more machine learning models for segmentation tasks, an application can invoke an inference service to execute machine learning models that perform one or more of the processing operations associated with the segmentation tasks.In at least one embodiment, the software 1318 implementing an advanced processing and inference pipeline, which includes a segmentation application and an anomaly detection application, can be streamlined because each application can call the same inference service to perform one or more inference tasks.

[0133] In at least one embodiment, the hardware 1322 can include GPUs, CPUs, graphics cards, an AI / deep learning system (e.g., an AI supercomputer such as NVIDIA's DGX), a cloud platform, or a combination thereof. In at least one embodiment, different types of hardware 1322 can be used to provide efficient, purpose-built support for software 1318 and services 1320 in the deployment system 1306. In at least one embodiment, the use of GPU processing can be implemented for on-premises processing (e.g., in the facility 1302), within an AI / deep learning system, in a cloud system, and / or in other processing components of the deployment system 1306 to improve the efficiency, accuracy, and effectiveness of image processing and generation.In at least one embodiment, the software 1318 and / or the services 1320 can be optimized for GPU processing with regard to deep learning, machine learning, and / or high-performance computing, as non-limiting examples. In at least one embodiment, at least part of the computing environment of the deployment system 1306 and / or training system 1304 can be run in a data center on one or more supercomputers or high-performance computers with GPU-optimized software (e.g., a hardware and software combination of NVIDIA's DGX system). In at least one embodiment, the hardware 1322 can include any number of GPUs that can be used to perform parallel data processing as described herein.In at least one embodiment, the cloud platform can also include GPU processing for GPU-optimized execution of deep learning tasks, machine learning tasks, or other computational tasks. In at least one embodiment, the cloud platform (e.g., NVIDIA's NGC) can run as a hardware abstraction and scaling platform using one or more AI / deep learning supercomputers and / or GPU-optimized software (e.g., as provided on NVIDIA's DGX systems). In at least one embodiment, the cloud platform can integrate an application container clustering or orchestration system (e.g., Kubernetes) across multiple GPUs to enable wireless scaling and load balancing.

[0134] Fig. Figure 14 is a system diagram for an exemplary system 1400 for generating and deploying an image processing deployment pipeline according to at least one embodiment. In at least one embodiment, the system 1400 can be used to implement the process 1300 of the Fig. 13 and / or other processes, including advanced processing and inference pipelines. In at least one embodiment, the system 1400 can include a training system 1304 and a deployment system 1306. In at least one embodiment, the training system 1304 and the deployment system 1306 can be implemented with software 1318, services 1320 and / or hardware 1322, as described herein.

[0135] In at least one embodiment, System 1400 (e.g., Training System 1304 and / or Deployment System 1306) can be implemented in a cloud computing environment (e.g., using Cloud 1426). In at least one embodiment, System 1400 can be implemented locally in relation to a healthcare facility or as a combination of cloud and local computing resources. In at least one embodiment, access to APIs in Cloud 1426 can be restricted to authorized users by defined security measures or protocols. In at least one embodiment, a security protocol can include web tokens that can be signed by an authentication service (e.g., AuthN, AuthZ, Gluecon, etc.) and appropriately authorized.In at least one embodiment, APIs of virtual instruments (described herein) or other instances of System 1400 can be restricted to a set of public IPs that have been verified or authorized for interaction.

[0136] In at least one embodiment, various components of System 1400 can communicate with each other over several of a variety of different network types, including, but not limited to, LANs (Local Area Networks) and / or WANs (Wide Area Networks), using wired and / or wireless communication protocols. In at least one embodiment, communication between devices and components of System 1400 (e.g., for transmitting inference requests, receiving results of inference requests, etc.) can take place via data bus(s), wireless data protocols (Wi-Fi), wired data protocols (e.g., Ethernet), etc.

[0137] In at least one embodiment, the training system 1304 training pipelines 1404 can be similar to those described herein with reference to Fig. as described in section 13. In at least one embodiment, in which one or more machine learning models are to be used in one or more deployment pipelines 1410 by the deployment system 1306, the training pipelines 1404 can be used to train or retrain one or more (e.g., pre-trained) models and / or to implement one or more pre-trained models 1406 (e.g., without the need for retraining or updating). In at least one embodiment, one or more output model(s) 1316 can be generated as a result of training pipelines 1404. In at least one embodiment, training pipelines 1404 can include any number of processing steps, such as, but not limited to, the conversion or adaptation of imaging data (or other input data).In at least one embodiment, different training pipelines 1404 can be used for different machine learning models used by the deployment system 1306. In at least one embodiment, a training pipeline 1404 can be similar to one referred to in . Fig. The first example described in section 13 for a first machine learning model can be used; a training pipeline 1404 can be similar to a second one with reference to Fig. The example described in section 13 can be used for a second machine learning model, and a training pipeline 1404 can be similar to one with reference to Fig. The third example described in Figure 13 can be used for a third machine learning model. In at least one embodiment, any combination of tasks within the training system 1304 can be used, depending on the requirements of each machine learning model. In at least one embodiment, one or more machine learning models can already be trained and ready for use, so that the machine learning models do not require processing by the training system 1304 and can be implemented by the deployment system 1306.

[0138] In at least one embodiment, one or more output models 1316 and / or pretrained models 1406 can include all types of machine learning models, depending on the implementation or embodiment. In at least one embodiment, and without limitation, machine learning models used by the system 1400 can include machine learning models that employ linear regression, logistic regression, decision trees, SVMs (Support Vector Machines), Naive Bayes, Knn (k-nearest neighbor), K-mean clustering, Random Forest, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autocoders, convolution, recurrence, perceptrons, long / short term memory (LSTM), Hopfield, Boltzmann, deep belief, deconvolutional, generative adversarial, liquid state machine, etc.), and / or other types of machine learning models.

[0139] In at least one embodiment, the training pipelines 1404 may include AI-supported annotation, as described herein with reference to at least Fig. 14 described in more detail. In at least one embodiment, labeled data 1312 (e.g., conventional annotations) can be generated by any number of techniques. In at least one embodiment, labels or other annotations can be generated in a drawing program (e.g., an annotation program), a computer-aided design (CAD) program, a labeling program, or another type of program suitable for generating annotations or labels for basic truth, and / or they can be drawn by hand in some examples. In at least one embodiment, basic truth data can be synthetically generated (e.g., from computer models or renderings), actually generated (e.g., designed and generated from real data), machine-automated (e.g., by using feature analysis and learning to extract features from data and then generate labels), or annotated by humans (e.g.,(a labeler or annotation expert defines the position of labels) and / or a combination thereof. In at least one embodiment, for each instance of imaging data 1308 (or other data types used by machine learning models), there can be corresponding ground truth data generated by the training system 1304. In at least one embodiment, the AI-assisted annotation can be performed as part of the deployment pipelines 1410; either in addition to or instead of the AI-assisted annotation included in the training pipelines 1404. In at least one embodiment, the system 1400 can include a multi-layered platform that can include a software layer (e.g., software 1318) of diagnostic applications (or other application types) that can perform one or more medical imaging and diagnostic functions. In at least one embodiment, the system 1400 can be communicatively (e.g.,(via encrypted links) with PACS server networks of one or more institutions. In at least one embodiment, the System 1400 can be configured to access and reference data from PACS servers in order to perform operations such as training machine learning models, deploying machine learning models, image processing, inference, and / or other operations.

[0140] In at least one embodiment, a software layer can be implemented as a secure, encrypted, and / or authenticated API through which applications or containers can be called from one or more external environments (e.g., facility 1302). In at least one embodiment, applications can then call or execute one or more services 1320 to perform computational, AI, or visualization tasks associated with the respective applications, and the software 1318 and / or the services 1320 can utilize the hardware 1322 for the effective and efficient execution of processing tasks. In at least one embodiment, communications sent to or received from a training system 1304 and a deployment system 1306 can be conducted using a pair of DICOM adapters 1402A, 1402B.

[0141] In at least one embodiment, the deployment system 1306 can execute deployment pipelines 1410. In at least one embodiment, deployment pipelines 1410 can include any number of applications that can be applied sequentially, non-sequentially, or otherwise to imaging data (and / or other data types) generated by imaging devices, sequencing devices, genomics devices, etc., including AI-assisted annotation as described above. In at least one of the embodiments described herein, a deployment pipeline 1410 for a single device can be referred to as a virtual instrument for a device (e.g., a virtual ultrasound instrument, a virtual CT scan instrument, a virtual sequencing instrument, etc.).In at least one embodiment, there can be more than one input pipeline 1410 for a single device, depending on what information is desired from data generated by a device. In at least one embodiment, there can be a first input pipeline 1410 if anomaly detection from an MRI machine is desired, and a second input pipeline 1410 if image enhancement from the output of an MRI machine is desired.

[0142] In at least one embodiment, an image generation application may include a processing task that involves the use of a machine learning model. In at least one embodiment, a user may wish to use their own machine learning model or select a machine learning model from model register 1324. In at least one embodiment, a user may implement their own machine learning model or select a machine learning model for inclusion in an application to perform a processing task. In at least one embodiment, applications may be selectable and customizable, and by defining constructs of applications, the deployment and implementation of applications for a particular user are presented as a more seamless user experience.In at least one embodiment, deployment pipelines 1410 can be made even more user-friendly, provide simpler integration and produce more accurate, efficient and timely results by utilizing other features of the system 1400 - such as services 1320 and hardware 1322.

[0143] In at least one embodiment, the deployment system 1306 may include a user interface (“UI”) 1414 (e.g., a graphical user interface, a web interface, etc.) that can be used to select applications for inclusion in the deployment pipeline(s) 1410, order applications, modify or change applications or parameters or constructs thereof, use and interact with the deployment pipeline(s) 1410 during setup and / or deployment, and / or otherwise interact with the deployment system 1306. In at least one embodiment, the user interface 1414 (or another user interface), although not shown with respect to the training system 1304, may be used to select models for use in the deployment system 1306, to select models for training or retraining in the training system 1304, and / or otherwise interact with the training system 1304.

[0144] In at least one embodiment, the pipeline manager 1412 can be used in addition to an application orchestration system 1428 to manage interaction between applications or containers of deployment pipeline(s) 1410 and services 1320 and / or hardware 1322. In at least one embodiment, the pipeline manager 1412 can be configured to facilitate application-to-application, application-to-service 1320, and / or application-or-service-to-hardware 1322 interactions. Although in at least one embodiment the pipeline manager 1412 is shown to be included in software 1318, this is not to be understood as a limitation, and in some examples the pipeline manager can be included in the services 1320. In at least one embodiment, the application orchestration system 1428 (e.g., Kubernetes, Docker, etc.) can be configured to facilitate application-to-application, application-to-service 1320, and / or application-to-service 1322 interactions.) include a container orchestration system that can group applications into containers as logic units for coordination, management, scaling, and deployment. In at least one embodiment, each application can be run with individual containers in a self-contained environment (e.g., at the kernel level) by associating applications from the deployment pipeline(s) 1410 (e.g., a reconstruction application, a segmentation application, etc.) to increase speed and efficiency.

[0145] In at least one embodiment, each application and / or container (or an image thereof) can be developed, modified, and deployed individually (e.g., a first user or developer can develop, modify, and deploy a first application, and a second user or developer can develop, modify, and deploy a second application separately from the first user or developer). This allows for the concentration and attention to be focused on a single application and / or container task without being hindered by tasks performed by one or more other applications or containers. In at least one embodiment, communication and cooperation between different containers or applications can be supported by the pipeline manager 1412 and the application orchestration system 1428.In at least one embodiment, as long as an expected input and / or output of each container or application is known to a system (e.g., based on constructs of applications or containers), the application orchestration system 1428 and / or the pipeline manager 1412 can facilitate communication between the individual applications or containers and the sharing of resources among them. In at least one embodiment, if one or more applications or containers in the deployment pipeline(s) 1410 can share the same services and resources, the application orchestration system 1428 can orchestrate, balance, and determine the sharing of services or resources between different applications or containers.In at least one embodiment, a scheduler can be used to track the resource requirements of applications or containers, the current or planned use of these resources, and resource availability. In at least one embodiment, a scheduler can thus allocate resources to different applications and distribute resources between applications with respect to system requirements and availability. In some examples, a scheduler (and / or another component of the application orchestration system 1428) can determine resource availability and distribution based on constraints imposed on a system (e.g., user constraints), such as QoS (Quality of Service), the urgency of data output needs (e.g., to determine whether real-time or delayed processing should be performed), and so on.

[0146] In at least one embodiment, the services 1320 used and shared by applications or containers in the deployment system 1306 may include computing services 1416, AI services 1418, visualization services 1420, and / or other service types. In at least one embodiment, applications may call (e.g., execute) one or more of the services 1320 to perform processing operations for an application. In at least one embodiment, the computing services 1416 may be used by applications to perform supercomputing or other high-performance computing (HPC) tasks. In at least one embodiment, the computing service(s) 1416 may be used to perform substantially simultaneous parallel processing (e.g., using a parallel computing platform 1430) for processing data by one or more applications and / or one or more tasks of a single application.In at least one embodiment, the Parallel Computing Platform 1430 (e.g., NVIDIA's CUDA) can enable general-purpose computing on GPUs (GPGPU) (e.g., GPUs / Graphics 1422). In at least one embodiment, a software layer of the Parallel Computing Platform 1430 can provide access to virtual instruction sets and parallel computing elements of GPUs for the execution of computing kernels. In at least one embodiment, the Parallel Computing Platform 1430 can include main memory, and in some embodiments, main memory can be shared between multiple containers and / or between different processing tasks within a single container.In at least one embodiment, inter-process communication (IPC) calls can be generated for multiple containers and / or for multiple processes within a container to use the same data from a shared memory segment of the Parallel Computing Platform 1430 (e.g., when multiple different stages of an application or multiple applications process the same information). In at least one embodiment, the same data can be used at the same location in memory for any number of processing tasks (e.g., at the same time, at different times, etc.), instead of creating a copy of the data and moving the data to different locations in memory (e.g., a read / write operation).In at least one embodiment, when data is used to generate new data as a result of processing, this information can be stored in a new data storage location and shared by different applications. In at least one embodiment, the storage location of data and a storage location for updated or modified data can be part of a definition of how a payload is to be understood within containers.

[0147] In at least one embodiment, the AI ​​services 1418 can be used to perform inference services for the execution of machine learning models assigned to applications (e.g., with the task of performing one or more processing tasks of an application). In at least one embodiment, the AI ​​services 1418 can utilize the AI ​​system 1424 to execute machine learning models (e.g., neural networks such as CNNs) for segmentation, reconstruction, object detection, feature detection, classification, and / or other inference tasks. In at least one embodiment, applications of the deployment pipeline(s) 1410 can use one or more output models 1316 from the training system 1304 and / or other application models to perform inference on image data. In at least one embodiment, two or more inference examples can be available using the application orchestration system 1428 (e.g., a scheduler).In at least one embodiment, a first category can include a high-priority / low-latency path that can achieve higher service level agreements, for example, for performing inference on urgent requests during an emergency or for a radiologist during diagnosis. In at least one embodiment, a second category can include a standard-priority path that can be used for requests that are not urgent or where the analysis can be performed at a later time. In at least one embodiment, the application orchestration system 1428 can distribute resources (e.g., services 1320 and / or hardware 1322) based on priority paths for different inference tasks of the AI ​​services 1418.

[0148] In at least one embodiment, the shared memory can be connected to the AI ​​services 1418 in the system 1400. In at least one embodiment, the shared memory can function as a cache (or another type of storage device) and be used to process inference requests from applications. In at least one embodiment, when an inference request is submitted, a request can be received from a set of API instances of the deployment system 1306, and one or more instances can be selected to process (e.g., for best match, load balancing, etc.) a request.In at least one embodiment, a request can be entered into a database to process a request; a machine learning model can be found in the model register 1324 if it is not already in a cache; a validation step can ensure that a suitable machine learning model is loaded into a cache (e.g., shared memory); and / or a copy of a model can be stored in a cache. In at least one embodiment, a scheduler (e.g., the pipeline manager 1412) can be used to start an application referenced in a request if an application is not already running or if there are not enough instances of an application. In at least one embodiment, an inference server can be started if no inference server has already been started to execute a model. Any number of inference servers can be started per model.In at least one embodiment, in a pull model where inference servers are clustered, data can be cached from the models whenever load balancing is advantageous. In at least one embodiment, inference servers can be statically loaded onto corresponding distributed servers.

[0149] In at least one embodiment, inference can be performed using an inference server running in a container. In at least one embodiment, an instance of an inference server can be associated with a model (and optionally multiple versions of a model). In at least one embodiment, if no instance of an inference server exists when a request to perform inference on a model is received, a new instance of an inference server can be loaded. In at least one embodiment, when an inference server is started, a model can be passed to an inference server, so that the same container can be used to serve different models, as long as the inference server runs as a different instance.

[0150] In at least one embodiment, an inference request for a specific application can be received during application execution, and a container (e.g., hosting an instance of an inference server) can be loaded (if not already loaded), and a startup procedure can be invoked. In at least one embodiment, preprocessing logic within a container can load incoming data, decode it, and / or perform additional preprocessing (e.g., using one or more CPUs and / or GPUs). In at least one embodiment, once the data is prepared for inference, a container can perform inference on the data as required. In at least one embodiment, this can involve a single inference call for a single image (e.g., a hand X-ray) or require inference for hundreds of images (e.g., a breast CT scan).In at least one embodiment, an application can summarize results before completion, which may include, without limitation, a single confidence score, pixel-level segmentation, voxel-level segmentation, generating a visualization, or generating text to summarize the results. In at least one embodiment, different models or applications may be assigned different priorities. For example, some models may have a real-time priority (TAT < 1 min), while others may have a lower priority (e.g., TAT < 10 min). In at least one embodiment, model execution times can be measured from the requesting institution or entity and may include the time spent traversing the partner network and execution by an inference service.

[0151] In at least one embodiment, requests between Services 1320 and inference applications can be hidden behind a Software Development Kit (SDK), and robust transport can be provided via a queue. In at least one embodiment, a request is placed in a queue via an API for a specific application / tenant ID combination, and an SDK pulls a request from the queue and passes it to an application. In at least one embodiment, the name of a queue can be specified in an environment from which an SDK retrieves it. In at least one embodiment, asynchronous communication via a queue can be beneficial because it allows each instance of an application to begin work as soon as it is available. Results can be transferred back via a queue to ensure that no data is lost.In at least one embodiment, queues can also provide the ability to segment work, since highest-priority work can go to a queue to which most instances of an application are connected, while lowest-priority work can go to a queue to which only a single instance is connected, processing tasks in the order they are received. In at least one embodiment, an application can run on a GPU-accelerated instance created in Cloud 1426, and an inference service can perform inference on a GPU.

[0152] In at least one embodiment, visualization services 1420 can be used to generate visualizations for viewing the outputs of applications and / or the deployment pipeline(s) 1410. In at least one embodiment, GPUs / graphics 1422 can be used by the visualization services 1420 to generate visualizations. In at least one embodiment, rendering effects, such as ray tracing, can be implemented by the visualization services 1420 to generate higher-quality visualizations. In at least one embodiment, visualizations can include, without limitation, 2D image renderings, 3D volume renderings, 3D volume reconstructions, 2D tomography layers, virtual reality displays, augmented reality displays, etc. In at least one embodiment, virtualized environments can be used to provide a virtual interactive display or environment (e.g.,to create a virtual environment for interaction by users of a system (e.g., doctors, nurses, radiologists, etc.). In at least one embodiment, the visualization services 1420 may include an internal visualizer, kinematics, and / or other rendering or image processing capabilities or functions (e.g., ray tracing, rasterization, internal optics, etc.).

[0153] In at least one embodiment, the hardware 1322 may include GPUs / Graphics 1422, the AI ​​system 1424, the cloud 1426, and / or any other hardware used to run the training system 1304 and / or the deployment system 1306. In at least one embodiment, the GPUs / Graphics 1422 (e.g., NVIDIA TESLA and / or QUADRO GPUs) may include any number of GPUs that can be used to perform processing tasks of compute services 1416, AI services 1418, visualization services 1420, other services, and / or any features or functions of the software 1318. Referring to AI services 1418, GPUs / Graphics s 1422 can be used, for example, to preprocess imaging data (or other data types used by machine learning models), to postprocess outputs of machine learning models and / or to perform inference (e.g., to run machine learning models).In at least one embodiment, the Cloud 1426, the AI ​​System 1424, and / or other components of the System 1400 can utilize GPUs / Graphics 1422. In at least one embodiment, the Cloud 1426 can include a GPU-optimized platform for deep learning tasks. In at least one embodiment, the AI ​​System 1424 can utilize GPUs, and the Cloud 1426—or at least a part of it tasked with deep learning or inference—can run using one or more AI Systems 1424. Although the Hardware 1322 is illustrated as discrete components, this is not to be understood as a limitation, and all components of the Hardware 1322 can be combined with or utilized by other components of the Hardware 1322.

[0154] In at least one embodiment, the AI ​​system 1424 can include a specially designed computing system (e.g., a supercomputer or an HPC) configured for inference, deep learning, machine learning, and / or other artificial intelligence tasks. In at least one embodiment, the AI ​​system 1424 (e.g., NVIDIA's DGX) can include GPU-optimized software (e.g., a software stack) that can be run using multiple GPUs / graphics 1422 in addition to CPUs, RAM, memory, and / or other components, features, or functions. In at least one embodiment, one or more AI systems 1424 can be deployed in the cloud 1426 (e.g., in a data center) to perform some or all of the AI-based processing tasks of the system 1400.

[0155] In at least one embodiment, Cloud 1426 can include a GPU-accelerated infrastructure (e.g., NVIDIA NGC) that can provide a GPU-optimized platform for performing processing tasks of System 1400. In at least one embodiment, Cloud 1426 can include one or more AI systems 1424 for performing one or more AI-based tasks of System 1400 (e.g., as a hardware abstraction and scaling platform). In at least one embodiment, Cloud 1426 can be integrated with the application orchestration system 1428, which utilizes multiple GPUs to enable seamless scaling and load balancing between applications and services 1320. In at least one embodiment, Cloud 1426 can be tasked with performing at least some of the services 1320 of System 1400, including compute services 1416, AI services 1418, and / or visualization services 1420, as described herein.In at least one embodiment, the Cloud 1426 can perform small and large batch inferences (e.g., running NVIDIA's TENSOR RT), provide an accelerated parallel computing API and platform 1430 (e.g., NVIDIA's CUDA), run an application orchestration system 1428 (e.g., KUBERNETES), provide a graphics rendering API and platform (e.g., for ray tracing, 2D graphics, 3D graphics, and / or other rendering techniques to generate higher-quality kinematics), and / or provide other functions for the System 1400.

[0156] Fig. Figure 15A illustrates a data flow diagram for a process 1500 for training, retraining, or updating a machine learning model according to at least one embodiment. In at least one embodiment, the process 1500 can be performed using the system 1400 of the Fig. 14 is performed as a non-restrictive example. In at least one embodiment, the process 1500 can utilize services and / or hardware as described herein. In at least one embodiment, refined models 1512 generated by the process 1500 can be executed by a deployment system for one or more containerized applications in deployment pipelines.

[0157] In at least one embodiment, model training 1514 can include retraining or updating an output model 1504 (e.g., a pre-trained model) using new training data (e.g., new input data, such as the customer record 1506, and / or new basic truth data associated with the input data). In at least one embodiment, retraining or updating the output model 1504 can involve resetting or deleting the output or loss layer(s) of the output model 1504 and / or replacing them with updated or new output or loss layer(s). In at least one embodiment, the output model 1504 can already have fine-tuned parameters (e.g., weights and / or biases) left over from previous training, so that training or retraining 1514 does not take as long or require as much processing as training a model from scratch.In at least one embodiment, during model training 1514, parameters can be updated and recalibrated for a new data set based on loss calculations associated with the accuracy of the output or loss layer(s) when generating predictions for a new customer data set 1506 by resetting or replacing the output or loss layer(s) of the output model 1504.

[0158] In at least one embodiment, the pre-trained models 1506 may be stored in a data storage device or a register. In at least one embodiment, the pre-trained models 1506 may have been trained, at least partially, in one or more facilities other than the facility executing the process 1500. In at least one embodiment, the pre-trained models 1506 may have been trained on-site using locally generated customer or patient data to protect the privacy and rights of patients, subjects, or customers of different facilities. In at least one embodiment, pre-trained models 1306 may be trained using a cloud and / or other hardware, but confidential, privacy-protected patient data must not be transferred to, used by, or accessible to components of a cloud (or other external hardware).In at least one embodiment, where a pre-trained Model 1506 is trained using patient data from more than one institution, the pre-trained Model 1506 may have been individually trained for each institution before being trained on patient or customer data from another institution. In at least one embodiment, such as when customer or patient data has been released due to privacy concerns (e.g., by a waiver, for experimental purposes, etc.), or when customer or patient data is included in a public dataset, customer or patient data from any number of institutions may be used to train the pre-trained Model 1506 on-premises and / or off-premises, such as in a data center or other cloud computing infrastructure.

[0159] In at least one embodiment, when selecting applications for use in deployment pipelines, a user can also select machine learning models for use with specific applications. In at least one embodiment, a user may not have a model available for use, allowing a user to select a pre-trained model for use with an application. In at least one embodiment, the pre-trained model may not be optimized to produce accurate results for a user's facility's customer dataset 1506 (e.g., based on patient diversity, demographic characteristics, types of medical imaging devices used, etc.).In at least one embodiment, the pre-trained model can be updated, retrained and / or fine-tuned for use in a specific facility before being deployed in a deployment pipeline for use with one or more applications.

[0160] In at least one embodiment, a user can select a pre-trained model to be updated, retrained, and / or fine-tuned, and this pre-trained model can be designated as the output model 1504 for a training system in the process 1500. In at least one embodiment, a customer dataset 1506 (e.g., imaging data, genomic data, sequencing data, or other data types generated by devices in a facility) can be used to perform model training (which, without limitation, may include transfer learning) on ​​the output model 1504 to produce a refined model 1512. In at least one embodiment, the basic truth data corresponding to the customer dataset 1506 can be generated by the training system 1304. In at least one embodiment, basic truth data can be generated, at least in part, by clinicians, scientists, physicians, or practitioners in a facility.

[0161] In at least one embodiment, AI-powered annotation can be used to generate basic truth data in some examples. In at least one embodiment, AI-powered annotation (e.g., implemented using an AI-powered annotation SDK) can utilize machine learning models (e.g., neural networks) to generate suggested or predicted basic truth data for a customer dataset. In at least one embodiment, a user can use annotation tools within a graphical user interface (GUI) on a computing device.

[0162] In at least one embodiment, the user 1510 can interact with a GUI via the computing device 1508 to edit or fine-tune (auto-)annotations. In at least one embodiment, a polygon editing feature can be used to move the vertices of a polygon to more accurate or finely tuned positions.

[0163] In at least one embodiment, once the customer dataset 1506 has associated basic truth data, basic truth data (e.g., from AI-assisted annotation, manual labeling, etc.) can be used during model training to generate a refined model 1512. In at least one embodiment, the customer dataset 1506 can be applied to the output model 1504 any number of times, and basic truth data can be used to update parameters of the output model 1504 until an acceptable level of accuracy is achieved for the refined model 1512. In at least one embodiment, once the refined model 1512 has been generated, it can be deployed in one or more deployment pipelines in a facility to perform one or more processing tasks related to medical imaging data.

[0164] In at least one embodiment, the refined model 1512 can be uploaded to a model register containing pre-trained models for selection by another facility. In at least one embodiment, this process can be completed in any number of facilities, so that the refined model 1512 can be further refined any number of times on new datasets to generate a more universal model.

[0165] Fig. Figure 15B is an exemplary illustration of a client-server architecture 1532 for improving annotation tools with pretrained annotation models according to at least one embodiment. In at least one embodiment, AI-supported annotation tools 1536 can be instantiated based on a client-server architecture 1532. In at least one embodiment, AI-supported annotation tools 1536 can assist radiologists in imaging applications, for example, in identifying organs and anomalies. In at least one embodiment, imaging applications can include software tools that help the user 1510, as a non-restrictive example, to identify some extreme points on a particular organ of interest in raw images 1534 (e.g., in a 3D MRI or CT scan) and to automatically obtain annotated results for all 2D slices of a particular organ.In at least one embodiment, results can be stored in a data store as training data 1538 and used as (e.g., and without limitation) basic truth data for training. In at least one embodiment, when the computing device 1508 sends extreme points for AI-assisted annotation, a deep learning model can receive this data as input and return inference results of a segmented organ or anomaly. In at least one embodiment, pre-instantiated annotation tools, such as the AI-assisted annotation tool 1536 in , can be used. Fig.15B, can be enhanced by API calls (e.g., API call 1544) to a server, such as an Annotation Assistant Server 1540, which may contain a set of pre-trained models 1542 stored, for example, in an annotation model registry. In at least one embodiment, an annotation model registry can store pre-trained models 1542 (e.g., machine learning models, such as deep learning models) that are pre-trained to perform AI-assisted annotation for a specific organ or anomaly. These models can be further updated using training pipelines. In at least one embodiment, pre-installed annotation tools can be enhanced over time as new labeled data is added.

[0166] The disclosure of this application also includes the following numbered clauses: Clause 1. At least one processor comprising: processing circuit arrangements for: generating, using at least one machine learning model, a first image corresponding to a foreground of an input image; generating, using the at least one machine learning model, one or more second images corresponding to a background of the input image; and causing a display of an animated effect generated using the first image and at least one of the one or more second images. Clause 2. At least one processor according to Clause 1, where the input image is a two-dimensional image. Clause 3. At least one processor according to a preceding clause, wherein the processing circuit arrangements shall further: generate a mask associated with the foreground of the first image; generate one or more replacement pixels for the mask using one or more trained neural networks. Clause 4. At least one processor according to a preceding clause, wherein the processing circuit arrangements shall further: identify one or more edge regions of a particular second image of the one or more second images; and generate one or more edge pixels using one or more trained neural networks. Clause 5. At least one processor according to a preceding clause, wherein the processing circuit arrangements shall further: determine one or more parameters assigned to a user; and select the animated effect based on the one or more parameters. Clause 6. At least one processor according to a preceding clause, wherein the processing circuit arrangements shall further: identify one or more components that are assigned to the foreground; and extract, from the input image, the one or more components. Clause 7. At least one processor according to Clause 6, wherein the processing circuit arrangements shall further: generate at least one second image of the one or more second images, which includes one or more second objects belonging to the one or more foreground components. Clause 8. At least one processor according to a preceding clause, wherein the processing circuit arrangements shall further: generate a configuration file associated with the animated effect, wherein the configuration file shall include at least one of the following: an effect, one or more effect parameters or identification information for the first image and at least one of the one or more second images. Clause 9. At least one processor according to any preceding clause, wherein the at least one processor consists of at least one of the following: a system for performing simulation operations; a system for performing simulation operations for testing or validating autonomous machine applications; a system for performing digital twinning operations; a system for performing light transport simulation; a system for rendering graphical output; a system for performing deep learning operations; a system implemented using an edge device; a system for generating or presenting virtual reality content (VR content); a system for generating or presenting augmented reality content (AR content); a system for generating or presenting mixed reality content (MR content); a system containing one or more virtual machines (VMs);a system for performing operations for a conversational AI application; a system for performing operations for a generative AI application; a system for performing operations using a language model; a system for performing one or more operations using a large language model (LLM); a system for performing one or more operations using a vision language model (VLM); a system that is at least partially implemented in a data center; a system that is implemented using a robot; a system for performing hardware tests using simulation; a system for performing one or more operations with generative content using a language model; a system for generating synthetic data; a collaborative content creation platform for 3D assets;or a system that is implemented at least partially using cloud computing resources. Clause 10. Computer-implemented method comprising: extracting one or more foreground components identified in an input image; generating, based on the input image, a first image that includes at least one portion of the one or more foreground components; determining a mask region associated with the one or more foreground components in the input image; generating one or more replacement pixels for the mask region; generating, based on the input image, at least one second image that includes the one or more replacement pixels in place of the mask region; and generating an animation using the first image and a special second image of the at least one second image. Clause 11. Computer-implemented method according to Clause 10, further comprising: identifying one or more foreground components in the input image based on a context of the input image. Clause 12. Computer-implemented method according to one of Clauses 10 or 11, further comprising: determining one or more settings associated with a user requesting access to one or more resources; and selecting at least one of the animation or properties of the animation based on the one or more settings. Clause 13. Computer-implemented method according to one of Clauses 10 to 12, wherein one or more of the input image, the first image and the at least one second image are two-dimensional images. Clause 14. Computer-implemented method according to any one of Clauses 10 to 13, further comprising: identifying one or more edges of a specific second image of the at least one second image; generating one or more edge pixels extending beyond the one edge or multiple edges of the specific second image; and generating, based on the input image, the particular second image that includes the one or more edge pixels. Clause 15. Computer-implemented method according to any of Clauses 10 to 14, wherein the one or more replacement pixels are generated using a trained neural network, and further comprising: receiving one or more prompts associated with the one or more replacement pixels. Clause 16. System comprising: one or more processors for generating an animated image for presentation during a waiting period for an application, using a first two-dimensional image (3D image) and at least one second 2D image, to generate the first 2D image from an input image associated with the application by extracting one or more foreground objects, and to generate the at least one second 2D image by inpainting a mask area of ​​the input image corresponding to the one or more foreground objects. Clause 17. System according to Clause 16, wherein the animated image is rendered on a client device during the waiting period based on a received set of configuration settings. Clause 18. System according to one of Clauses 16 or 17, wherein the one or more processors are further to receive one or more input prompts for conditioning a trained neural network to perform inpainting. Clause 19. System according to one of Clauses 16 to 18, wherein the one or more processors shall further perform outpainting of additional edge pixels for the at least one second 2D image. Clause 20. System according to any of Clauses 16 to 19, wherein the system consists of at least one of the following: a system for performing simulation operations; a system for performing simulation operations for testing or validating autonomous machine applications; a system for performing digital twinning operations; a system for performing light transport simulation; a system for rendering graphical output; a system for performing deep learning operations; a system implemented using an edge device; a system for generating or presenting virtual reality content (VR content); a system for generating or presenting augmented reality content (AR content); a system for generating or presenting mixed reality content (MR content); a system containing one or more virtual machines (VMs); a system for performing operations for a conversational AI application;a system for performing operations for a generative AI application; a system for performing operations using a language model; a system for performing one or more operations using a large language model (LLM); a system for performing one or more operations using a vision language model (VLM); a system that is at least partially implemented in a data center; a system that is implemented using a robot; a system for performing hardware tests using simulation; a system for performing one or more operations with generative content using a language model; a system for generating synthetic data; a collaborative content creation platform for 3D assets; or a system that is at least partially implemented using cloud computing resources.

[0167] Other variations are within the spirit of the present disclosure. While disclosed techniques may be subject to various modifications and alternative constructions, certain illustrated embodiments are shown in the drawings and have been described in detail above. It must be understood, however, that the disclosure is not intended to be limited to any specific form or forms disclosed; on the contrary, it is intended to cover all modifications, alternative constructions, and equivalents that fall within the meaning and scope of the disclosure, as defined in the attached claims.

[0168] The use of "a" and "the" and similar references in connection with describing disclosed embodiments (particularly in connection with the following claims) must be interpreted as encompassing both the singular and the plural, unless otherwise stated herein or clearly contradicted by the context. The terms "comprising," "exhibiting," "including," and "containing" are to be interpreted as open terms (that is, meaning "inclusive but not limited to"), unless otherwise stated. The term "connected," when used unchanged and referring to physical connections, must be interpreted as partially or wholly contained in, attached to, or connected to one another, even if something intervenes in between.The specification of value ranges is intended solely as a quick method of referring individually to each separate value falling within the range, unless otherwise stated herein, and each separate value is included in the specification as if it were mentioned individually. The use of the terms "set" (e.g., "a set of elements") or "subset," unless otherwise noted herein or contradicted by the context, is to be understood as a non-empty collection comprising one or more elements. Unless otherwise noted or contradicted by the context, the term "subset" of a corresponding set does not necessarily denote a proper subset of the corresponding set; rather, subset and corresponding set may be the same.

[0169] Conjunctive expressions, such as sentences of the form "at least one of A, B, and C" or "at least one of A, B, and C," are generally understood, unless explicitly stated otherwise or clearly contradicted by the context, to mean that an element, term, etc., can be either A, B, or C, or a non-empty subset of a set of A, B, and C; either A, B, or C; or any non-empty subset of a set of A, B, and C. Thus, in the illustrative example of a set of elements, the conjunctive sentences "at least one of A, B, and C" and "at least one of A, B, and C" refer to one of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. The conjunctive expression is therefore not generally intended to mean that in certain embodiments, at least one of A, at least one of B, and at least one of C must be present.Additionally, unless otherwise noted or contradicted by the context, the term "plurality" indicates a state of plurality (e.g., "a multitude of elements" indicates multiple elements). A multitude consists of at least two elements, but there may be more if this is either explicitly stated or indicated by context. Unless otherwise noted or clearly contradicted by the context, the phrase "based on" means "at least partly based on" and not "exclusively based on".

[0170] All operations of processes described herein may be performed in a suitable order unless otherwise specified herein or the context clearly contradicts this. In at least one embodiment, a process such as those described herein (or variations and / or combinations thereof) is carried out under the control of one or more computer systems configured with executable instructions and is implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that runs collectively on one or more processors, by hardware or combinations thereof. In at least one embodiment, code is stored on a computer-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions that can be executed by one or more processors.In at least one embodiment, a computer-readable storage medium is a non-transitory computer-readable storage medium that excludes transitory signals (e.g., a propagating transient electrical or electromagnetic transmission) but includes non-transitory data storage circuitry (e.g., buffers, caches, and queues) within transceivers for transitory signals. In at least one embodiment, code (e.g., executable code or source code) is stored on a set of one or more non-transitory, computer-readable storage media containing executable instructions (or other memory for storing executable instructions) which, when executed by one or more processors of a computer system (i.e., as a result of execution), cause the computer system to perform the operations described herein.In at least one embodiment, a set of non-transitory computer-readable storage media comprises multiple non-transitory computer-readable storage media. One or more individual non-transitory storage media within a set of multiple non-transitory computer-readable storage media lack all the code, while multiple non-transitory computer-readable storage media collectively store all the code. In at least one embodiment, executable instructions are executed such that different instructions are executed by different processors. For example, a non-transitory computer-readable storage medium stores instructions, and a main CPU executes some of the instructions, while a graphics processing unit (GPU) executes other instructions.In at least one embodiment, different components of the computer system have separate processors, and different processors execute different subsets of instructions.

[0171] Accordingly, in at least one embodiment, computer systems are configured to implement one or more services that individually or collectively perform operations of processes described herein, and such computer systems are configured with applicable hardware and / or software that enables the performance of operations. Furthermore, a computer system implementing at least one embodiment of the present disclosure is a single device, and in other embodiments, it is a distributed computer system comprising several devices that operate differently, such that the distributed computer system performs operations described herein, and such that no single device performs all operations.

[0172] The use of any and all examples or exemplary language (e.g., "such as") provided herein is intended only to better illustrate the embodiments of the disclosure and does not limit the scope of protection of the invention unless otherwise claimed. No expression in the specification should be interpreted as indicating any unclaimed elements as essential to the implementation of the disclosure.

[0173] All references, including publications, patent applications and patents cited herein, are hereby incorporated by reference to the same extent as if each reference had been individually and specifically indicated to be incorporated by reference and set forth herein in full.

[0174] In the description and claims, the terms "coupled" and "connected," along with their derivatives, may be used together. It is understood that these terms are not intended as synonyms. In specific examples, "connected" or "coupled" may instead be used to indicate that two or more elements are in direct physical or electrical contact with each other. However, "coupled" can also mean that two or more elements are not in direct contact with each other, but nevertheless work together or interact.

[0175] Unless specifically stated otherwise herein, it is clear that terms such as "processing", "calculating", "calculating", "determining" or the like denote the action and / or processes of a computer or computational system or similar electronic computing devices that process data and / or convert data represented as physical quantities, such as electronically, within the registers of the computing system and / or memory into other data represented similarly as physical quantities within the memory, registers or other data storage, transmission or display devices of the computing system.

[0176] The term "processor" can similarly refer to any device or part of a device that processes electronic data from registers and / or memory to transform the electronic data into other electronic data that can be stored in registers and / or memory. As non-restrictive examples, a "processor" can be a CPU or a GPU. A "computing platform" can include one or more processors. As used herein, "software" processes can, for example, include software and / or hardware entities that perform work, such as tasks, threads, and intelligent agents, over time. Each process can therefore refer to multiple processes for executing instructions sequentially or in parallel, continuously or intermittently.The terms “system” and “procedure” are used interchangeably here insofar as a system can embody one or more procedures, and procedures can be considered as a system.

[0177] This document may refer to the acquisition, collection, reception, or input of analog or digital data into a subsystem, computer system, or computer-implemented machine. The acquisition, collection, reception, or input of analog or digital data can be accomplished in a variety of ways, such as receiving data as a parameter of a function call or an application programming interface (API) call. In some implementations, the process of acquiring, collection, reception, or inputting analog or digital data may involve transferring data over a serial or parallel interface. In other implementations, the process of acquiring, collection, reception, or inputting analog or digital data may involve transferring data over a computer network from a providing entity to a receiving entity.It can also refer to the provision, output, transmission, sending, or presentation of analog or digital data. In various examples, processes of providing, outputting, transmission, sending, or presenting analog or digital data can be accomplished by transferring data as an input or output parameter of a function call in an application programming interface or an interprocess communication mechanism.

[0178] Although the above explanation presents exemplary implementations of the described techniques, other architectures may be used to implement the described functionality, and they are intended to be within the scope of this disclosure. Furthermore, although specific distributions of responsibilities are defined above for illustrative purposes, various functions and responsibilities may be distributed and divided differently depending on the circumstances.

[0179] Although the subject matter was described in terms specific to structural features and / or methodological actions, this does not mean that the subject matter defined in the appended claims is necessarily limited to the specific features or actions described. Rather, the specific features and actions are disclosed as exemplary forms of implementing the claims.

[0180] It is understood that the aspects and embodiments described above are purely exemplary, and that modifications of details may be made within the scope of protection of the claims.

[0181] Each device, each method and each feature disclosed in the description, and (where applicable) the claims and drawings, may be provided independently or in any suitable combination.

[0182] Reference numerals appearing in the claims are for illustrative purposes only and do not restrict the scope of protection of the claims.

Claims

[1] At least one processor comprising the following: Processing circuit arrangements for: Generate, using at least one machine learning model, a first image that corresponds to a foreground of an input image; Generate, using at least one machine learning model, one or more secondary images that correspond to a background of the input image; and Causing the display of an animated effect created using the first image and at least one of the one or more second images. [2] At least one processor according to claim 1, wherein the input image is a two-dimensional image. [3] At least one processor according to any of the preceding claims, wherein the processing circuit arrangements shall further comprise: Creating a mask that is assigned to the foreground of the first image; Generating one or more replacement pixels for the mask using one or more trained neural networks. [4] At least one processor according to any of the preceding claims, wherein the processing circuit arrangements shall further comprise: Identifying one or more edge regions of a particular second image of one or more second images; and Generating one or more edge pixels using one or more trained neural networks. [5] At least one processor according to the preceding claim, wherein the processing circuit arrangements shall further comprise: Determining one or more parameters assigned to a user; and Selecting the animated effect based on one or more parameters. [6] At least one processor according to the preceding claim, wherein the processing circuit arrangements shall further comprise: Identifying one or more components that are assigned to the foreground; and Extracting one or more components from the input image. [7] At least one processor according to claim 6, wherein the processing circuit arrangements shall further comprise: Generating at least one second image of the one or more second images, which includes one or more second objects belonging to the one or more components in the foreground. [8] At least one processor according to the foregoing claim, wherein the processing circuit arrangements shall further comprise: Generating a configuration file associated with the animated effect, wherein the configuration file includes at least one of the following: an effect, one or more effect parameters or identification information for the first image and at least one of the one or more second images. [9] At least one processor according to any of the preceding claims, wherein the at least one processor consists of at least one of the following: a system for performing simulation operations; a system for performing simulation operations to test or validate autonomous machine applications; a system for conducting digital twinning operations; a system for performing light transport simulation; a system for rendering graphical output; a system for performing deep learning operations; a system that is implemented using an edge device; a system for creating or presenting virtual reality content (VR content); a system for creating or presenting augmented reality content (AR content); a system for creating or presenting mixed-reality content (MR content); a system that contains one or more virtual machines (VMs); a system for performing operations for a conversational AI application; a system for performing operations for a generative AI application; a system for performing operations using a language model; a system for performing one or more operations using a large language model (LLM); a system for performing one or more operations using a Vision Language Model (VLM); a system that is at least partially implemented in a data center; a system that is implemented using a robot; a system for performing hardware tests using simulation; a system for performing one or more operations with generative content using a language model; a system for generating synthetic data; a collaborative content creation platform for 3D assets; or a system that is implemented at least partially using cloud computing resources. [10] Computer-implemented method comprising the following: Extracting one or more foreground components identified in an input image; Generate, based on the input image, a first image that includes at least one section of one or more foreground components; Defining a mask area that is assigned to one or more foreground components in the input image; Generating one or more replacement pixels for the mask area; Generate, based on the input image, at least one second image that includes the one or more replacement pixels in place of the mask area; and Creating an animation using the first image and a special second image, or at least one second image. [11] Computer-implemented method according to claim 10, further comprising: Identifying one or more foreground components in the input image based on the context of the input image. [12] Computer-implemented method according to one of claims 10 or 11, further comprising: Determine one or more settings that are assigned to a user requesting access to one or more resources; and Select at least one of the animations or animation properties based on one or more settings. [13] Computer-implemented method according to one of claims 10 to 12, wherein one or more of the input image, the first image and the at least one second image are two-dimensional images. [14] Computer-implemented method according to any one of claims 10 to 13, further comprising: Identifying one or more edges of a specific second image of at least one second image; Generating one or more edge pixels that extend beyond the one or more edges of the specific second image; and Generate, based on the input image, the special second image that includes one or more edge pixels. [15] Computer-implemented method according to any one of claims 10 to 14, wherein the one or more replacement pixels are generated using a trained neural network, and further comprising: Receiving one or more prompts associated with the one or more replacement pixels. [16] System, encompassing: one or more processors for generating an animated image for presentation during a waiting period for an application using a first two-dimensional image (2D image) and at least one second 2D image, to generate the first 2D image from an input image associated with the application by extracting one or more foreground objects, and to generate the at least one second 2D image by inpainting a mask area of ​​the input image corresponding to the one or more foreground objects. [17] System according to claim 16, wherein the animated image is rendered on a client device during the waiting period based on a received set of configuration settings. [18] System according to one of claims 16 or 17, wherein the one or more processors shall further receive one or more input prompts for conditioning the trained neural network to perform inpainting. [19] System according to one of claims 16 to 18, wherein one or more processors are further said to perform outpainting of additional edge pixels for the at least one second 2D image. [20] System according to any one of claims 16 to 19, wherein the system consists of at least one of the following: a system for performing simulation operations; a system for performing simulation operations to test or validate autonomous machine applications; a system for conducting digital twinning operations; a system for performing light transport simulation; a system for rendering graphical output; a system for performing deep learning operations; a system that is implemented using an edge device; a system for creating or presenting virtual reality content (VR content); a system for creating or presenting augmented reality content (AR content); a system for creating or presenting mixed-reality content (MR content); a system that contains one or more virtual machines (VMs); a system for performing operations for a conversational AI application; a system for performing operations for a generative AI application; a system for performing operations using a language model; a system for performing one or more operations using a large language model (LLM); a system for performing one or more operations using a Vision Language Model (VLM); a system that is at least partially implemented in a data center; a system that is implemented using a robot; a system for performing hardware tests using simulation; a system for performing one or more operations with generative content using a language model; a system for generating synthetic data; a collaborative content creation platform for 3D assets; or a system that is implemented at least partially using cloud computing resources. 97

Images