Preloading software applications in a cloud computing environment

Abstract

Apparatus, systems, and techniques for preloading software applications in a cloud computing environment. The method may include sending a preload request to preload a first portion of data for an application hosted in an application hosting platform, the preload request being received prior to receiving user input identifying the application to execute. The method may include receiving a first indication that the first portion of data has been preloaded and receiving a user request to execute the application. The method may further include sending a load request to load a second portion of data for the application, receiving a second indication that the second portion of data has been loaded for the application, and executing the application in the virtualized computing environment in response to receiving the second indication.

Description

[Technical field]

[0001] At least one embodiment relates to processing resources used to perform and facilitate preloading of applications in a cloud computing environment. For example, at least one embodiment relates to a processor or computing system used to provide and enable preloading of gaming applications in one or more virtualized computing environments hosted by an application hosting platform in accordance with various novel techniques described herein. [Background technology]

[0002] In a cloud computing environment, a user may access and stream a software application (e.g., a gaming application) through an application hosting platform on his / her local client device. The software application may load several files, such as assets, textures, graphics, data associated with the user, during the startup sequence. Loading all the files associated with the software application may take a significant amount of time. The application hosting platform may load the software application when the user accesses an instance of the software application. However, these existing techniques may make the user wait a significant amount of time until the software application is ready for streaming. This may result in a degraded overall user experience. Summary of the Invention [Means for solving the problem]

[0003] Various embodiments according to the present disclosure will now be described with reference to the drawings. [Brief description of the drawings]

[0004] [Figure 1A] FIG. 1 is a block diagram of an example cloud computing environment system architecture according to at least some embodiments. [Figure 1B] FIG. 11 is a block diagram of another example system architecture, according to at least some embodiments. [Diagram 2] FIG. 1 is a block diagram of a cloud environment, according to at least some embodiments. [Diagram 3] FIG. 1 illustrates a diagram of an example of preloading software applications, according to at least some embodiments. [Figure 4] FIG. 1 shows an example diagram of preloading software applications, according to at least some embodiments. [Diagram 5] FIG. 1 illustrates a diagram of an example of preloading software applications, according to at least some embodiments. [Figure 6] FIG. 1 is a flow diagram of an example method for preloading software applications, according to at least some embodiments. [Figure 7A] FIG. 1 illustrates inference and / or training logic, according to at least some embodiments. [Figure 7B] FIG. 1 illustrates inference and / or training logic, according to at least some embodiments. [Figure 8] FIG. 1 illustrates an example data center system according to at least some embodiments. [Figure 9] FIG. 1 illustrates a computer system according to at least some embodiments. [Figure 10] FIG. 1 illustrates a computer system according to at least some embodiments. [Figure 11] A diagram illustrating at least a portion of a graphics processor according to at least some embodiments. [Figure 12]A diagram illustrating at least a portion of a graphics processor according to at least some embodiments. [Figure 13] FIG. 1 illustrates an example data flow diagram for an advanced computing pipeline, according to at least some embodiments. [Figure 14] A system diagram of an example system for training, adapting, instantiating, and deploying machine learning models in an advanced computing pipeline, according to at least some embodiments. [Figure 15A] FIG. 1 is a data flow diagram of a process for training a machine learning model, according to at least some embodiments. [Figure 15B] FIG. 1 illustrates a client-server architecture for extending annotation tools with pre-trained annotation models, according to at least some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005] A software application may be hosted on a cloud computing environment, for example, without limitation, by an application hosting platform. For example, an instance of a software application hosted by the application hosting platform may execute on a client device via a virtualized computing environment (e.g., on a virtual machine or virtual container). A user may access a software application (e.g., a video game or a mobile game) on a computing device (e.g., a computer, a gaming console, a mobile phone, a smart phone, etc.) via the application hosting platform. For example, a user may access (e.g., log in to) the application hosting platform and select a game application to run or play on his or her computing device. The application hosting platform may communicate with the game application via a set of application programming interfaces (APIs) and begin loading an instance of the game application into the virtualized computing environment for access by the user device. For example, the application hosting platform may launch the game application into the virtualized computing environment using command line interface (CLI) commands once the user selects the game application. To launch the game application in the virtualized computing environment, the application hosting platform can begin loading a game application file for the user in the virtualized computing environment. In some instances, the game application file can include general game application data and user game application data.For example, generic data for a game application may include the game application's assets, textures, artwork, background graphics, title screen, main menu screen, the game application's memory allocation for the user device, shaders, etc. User data may include data that the game application loads specifically for a user. For example, user data may include specific user settings (e.g., resolution and shader settings), custom user content (e.g., customizations for an in-game character created by the user), custom key bindings, etc.

[0006] In some instances, it may take several minutes for the application hosting platform to load the general data and user data into the virtualized computing environment. In particular, for example, if a large portion of the data loaded by the application hosting platform from a game application is general data, it may take several minutes for the application hosting platform to load the general data. During this time, the user may not be able to access the game application and may be forced to wait. In some instances, the user may see a loading screen while the virtualized computing environment prepares for play. Once initialization is complete, the virtualized computing environment may be ready and the user may access the game application and the main menu. The overall user experience of running an instance of a game application may be adversely affected by a long delay between when the user selects the game application and when the user can access the game application. For example, the experience may be adversely affected if the user must wait a long time for the virtualized computing environment to sufficiently complete initialization and loading of general data and user data before the game can be played.

[0007] Aspects of the present disclosure address these and other deficiencies by enabling an application hosting platform to prepare (e.g., preload at least some portions of) a virtualized computing environment (e.g., a virtual machine or virtual container) for executing an instance of a software application, thereby reducing the time between selecting an instance of the software application and executing the instance of the software application for a user. That is, the application hosting platform can cause an instance of the software application to be at least partially loaded (using a preload mode) when a user selects an instance of the software application, e.g., general data of the software application is partially loaded into the virtualized computing environment when the user selects the software application to run. For example, the software application can send an initialization request to the application hosting platform via a first application programming interface (API) command. The initialization request can inform the application hosting platform whether the software application should be instructed to launch in preload mode using a CLI command or a flag setting (e.g., in the application's configuration file). Alternatively or additionally, the software application should be instructed to launch in preload mode by verifying that the software application exists within the cloud environment.

[0008] The application hosting platform may then request that general data for the instance of the software application be loaded into the virtualized computing environment. The software application may begin loading general data into the virtualized computing environment for the instance of the software application in response to the confirmation. For example, the virtualized computing environment may be loaded with textures, shaders, background graphics, etc., after receiving the confirmation, e.g., everything except user data. Each software application may have a varying amount of general data loaded, e.g., the general data may be specific to each software application. In some instances, the software application may notify the application hosting platform that the general data loading is complete after loading the general data. In some instances, the application hosting platform may utilize machine learning models to predict when a particular software application will finish loading the general data. Once the application hosting platform receives the notification that the software application has finished loading the general data, it may place the virtualized computing environment in an idle mode. That is, the virtualized computing environment may be ready for a user to access while consuming minimal computing resources.

[0009] When the user selects an instance of the software application, the virtualized computing environment service can notify the application hosting platform that the user has initialized a session. The application hosting platform can communicate with the application via an API command (e.g., a session callback) to instruct the software application to load user data. Thus, the virtualized computing environment can be loaded with user data and ready for the user to begin executing the software application. The application hosting platform can request to preload multiple virtualized computing environments for a particular software application based on the collected data. For example, the application hosting platform can determine the average duration that the virtualized computing environments are idle and the number of users who do not have access to the preloaded virtualized computing environments, and can determine, for example, that there are not enough preloaded virtual computing environments for the users of the software application. Thus, based on a balance between the amount of time that the virtualized computing environments are idle and the number of users who do not have access to the preloaded virtualized computing environments, the application hosting system can instruct to prepare a certain amount of virtualized computing environments for the software (e.g., gaming, content creation, etc.) application. Additionally, when a software application is updated, the application hosting platform may be instructed to deconstruct the virtualized computing environment, for example, by deleting the virtual machine and pre-loading another virtual machine with updated general data from the updated software application (or replacing the general data stored in the virtual machine).

[0010] Because the general data is already loaded in the virtualized computing environment, the amount of time a user must wait between starting a software application and running the software application is reduced. For example, instead of waiting until the general data and user data are loaded, a user can access the main menu and start the software application after the user data is loaded. This improves the overall user experience and allows for a more balanced use of network bandwidth and computing resources.

[0011] 1 illustrates a block diagram of an example system architecture 100, according to at least one embodiment. The system architecture 100 (also referred to herein as the "system") includes an application hosting platform 102, an application developer platform 104, a server machine 106, client devices 108A-108N (collectively and individually referred to as client devices 108), and a data store 112, each of which is connected to a network 120. In an implementation, the network 120 may include a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or a wide area network (WAN)), a wired network (e.g., an Ethernet network), a wireless network (e.g., an 802.11 network or a Wi-Fi network), a cellular network (e.g., a Long Term Evolution (LTE) network), a router, a hub, a switch, a server computer, and / or combinations thereof.

[0012] In some implementations, the data store 112 is a persistent storage capable of storing content items and data structures for tagging, organizing, and indexing the content items. The data store 112 may be hosted by one or more storage devices, such as main memory, magnetic or optical storage based disks, tapes or hard drives, NAS, SAN, etc. In some implementations, the data store 112 may be a file server connected to a network, and in other embodiments, the data store 112 may be some other type of persistent storage, such as an object-oriented database, a relational database, etc., that may be hosted by the platform 102 or one or more different machines coupled to the platform 102 via the network 120.

[0013] The application hosting platform 102 may be configured to host files of one or more applications (e.g., application 130A, application 130B, etc.) provided by an application developer (e.g., via the application developer platform 104). The application developer platform 104 may be used by an application developer (e.g., a user, a company, an organization, etc.). For example, the application developer may be a video game developer developing a video game (represented by application 130) for users to interact with on client devices 108. The application hosting platform 102 may provide users, via their respective client devices 108A-108N, access to the applications 130 (or instances of applications 130) provided by the application developer platform 104. For example, the application hosting platform 102 may enable users to consume, upload, download, and / or search for the applications 130. In some embodiments, the application hosting platform 102 may have a website (e.g., one or more web pages) or a client application or client component that may be used to provide users with access to the applications 130. In some embodiments, each application 130 may be comprised of general data 132 (e.g., data excluding user data) and user data 134, e.g., general data 132A and user data 134A for application 130A, general data 132B and user data 134B for application 130B, etc.In at least one embodiment, application hosting platform 102 may be an example of a cloud-hosted gaming services platform, a cloud-hosted collaborative content creation platform for disparate content creation applications, a video streaming hosting platform, a testing platform for simulated or augmented content, a machine learning training platform, a machine learning deployment platform, or a video conferencing hosting platform. In at least one embodiment, application 130 (or an instance of application 130) may be an example of a gaming application, a video conferencing application, a content creation application, a cloud-hosted application, a collaborative content creation application, a cloud-hosted collaborative content creation application, a video streaming application, a machine learning application, or a simulation application.

[0014] In some embodiments, the server 106 may host a virtualized computing environment 160 that executes an instance of the application 130. For example, the server 106 may be a computer system that includes one or more physical devices (e.g., a processing device (e.g., a GPU), memory, one or more I / O devices, etc.) and a hypervisor and / or host operating system that manages one or more virtualized computing environments 160. The virtualized computing environment 160 may correspond to, for example, a virtual machine that executes a guest operating system and one or more guest applications including an instance of the application 130, or a container that executes an application, such as an instance of the application 130. One or more servers 106 may be provided, and each server 106 may host one or more virtualized computing environments 160. In some embodiments, each server 106 may correspond to a computer system 900 and / or a computer system 1000 described with respect to FIGS. 9 and 10.

[0015] The application hosting platform 102 may include an application load manager 150, as described in more detail herein, that incrementally loads application data (general data 132A and user data 134A) into the virtualized computing environment 160 to reduce the time between selecting an application 130A or 130B and executing an instance of the application 130A or 130B for a user. At least some of the general data 132A and user data 134A may be displayed via a user interface (UI) on each client device that accesses the respective application 130 instance. The amount of general data 132 and user data 134 may be defined for the application 130 by the application developer via the application developer platform 104. A user of each client device 108 may interact with the application 130 instance by engaging with the application 130 instance that has the general data 132 and user data 134 loaded therein via a GUI provided by the application hosting platform 102. The GUI of the application hosting platform 102 may be presented by a client component of the application hosting platform 102 or may be rendered by a web browser hosted by the client device 108.

[0016] According to embodiments provided herein, the virtualized computing environment 160 may be instantiated to facilitate execution of an instance of an application 130 for access by a client device 108 and may be torn down following termination of the application instance requested by a user of the client device 108. For example, a user accessing an application instance via an application hosting platform GUI on the client device 108 may log out of an account associated with the application 130 hosted by the application hosting platform 102. Upon detecting that the user has “logged out” of the associated account, the application load manager 150 may send a teardown request to the server 106 that may cause the hypervisor to tear down the virtualized computing environment 160.

[0017] A user of each client device 108 may engage (e.g., consume, interact, etc.) with the application 130 (e.g., via an application hosting platform GUI) that is loaded with the general data 132 and the user data 134 to progress through the application via the respective client device 108. In an illustrative example, the applications 130A and 130B may be video game applications (e.g., game applications) developed by a video game developer. In another illustrative example, the applications 130A and 130B may be content creation applications or asset creation applications of a cloud-hosted collaborative content creation platform. The general data 132 and user data 134 of each video game application 130 may be presented in the application hosting platform GUI on the client device 108 for consumption by the user of the client device 108. In some embodiments, the general data 132 may include application 130 assets, textures, artwork, background graphics, title screens, main menu screens, memory allocations of the game application 130 for the client devices 108, shaders, etc. In some embodiments, the user data 134 may include data that the application 130 loads specifically for each client device 108 that selects the game application 130, e.g., for a particular user. For example, the user data 134 may include specific user settings (e.g., resolution or shader settings), custom user content (e.g., customizations for an in-game character created by the user), custom key bindings, etc. Thus, a user may customize the experience of running the game application 130 on each client device 108.

[0018] Client devices 108 may include devices including, but not limited to, televisions, smart phones, mobile phones, personal digital assistants (PDAs), portable media players, netbooks, laptop computers, e-book readers, tablet computers, desktop computers, set-top boxes, gaming consoles, etc. As mentioned above, each client device 108 may include a client component of the application hosting platform 102 (or a web browser) that provides a GUI that allows a user of the client device 108 to request execution of an application 130. The GUI may provide rendered versions of general data 132 and user data 134 for presentation during run-time of the application 130 and may allow a user to provide input during run-time of the application 130.

[0019] In some embodiments, server machine 106 may be separate from the serve machines that support application hosting platform 102. In other embodiments, server machine 106 may be part of application hosting platform 102. In some embodiments, one or more server machines 106, application hosting platform 102, application developer platform 104, and data store 112 may be part of a cloud environment that is accessible by client devices 108A-108N over network 120.

[0020] FIG. 2 illustrates a block diagram of a cloud environment 200, according to at least one embodiment. In some embodiments, the cloud environment 200 includes an application hosting platform 102, a sign-on platform 250, and one or more servers 106, each of which provides one or more virtual computing environments 160. Client devices 108 are connected to the cloud environment via a network 120 as described with reference to FIG. 1. In some embodiments, the sign-on platform 250 can include an account manager 205, an account linking component 215, and an identity manager 210. The application hosting platform 102 can include an application load manager 150, which includes a scheduler 225 and a software application service 230. The application hosting platform 102 can also include a game application engine 240 and an application file repository 260.

[0021] In some embodiments, the account linking component 215 can link an account of a user of a client device 108 to an account associated with the application hosting platform 102. In some embodiments, the account linking component 215 can also determine whether an account of a user of a client device 108 associated with an application (e.g., application 130 described with reference to FIG. 1 ) is linked or whether the user is “logged in”. In at least one embodiment, if a user’s account associated with an application 130 is not linked, the account linking component 215 can facilitate account linking. In at least one embodiment, if a user’s account associated with an application 130 is linked, the account linking component 215 can store respective credentials or access tokens and refresh tokens (e.g., tokens used to grant a user access to the application 130). In some embodiments, the account linking component 215 can link or maintain many different user’s credentials for many applications 130 hosted by the application hosting platform 130.

[0022] In at least one embodiment, the account manager 205 can manage accounts associated with the application 130 for users of the client devices 108. For example, the account manager 205 can provide a UI that allows users of the client devices 108 to log in to accounts associated with the application 130.

[0023] In at least one embodiment, the sign-on platform 250 may include a unique account manager 205 for each unique application 130 hosted on the application hosting platform 130. In some embodiments, the identity manager 210 may validate (e.g., check) to ensure that a user's credentials are associated with an account for the application 130. That is, the identity manager 210 may allow a user to access their account associated with the application 130.

[0024] The application hosting platform 102 may include an application load manager 150 that may enable incremental loading of instances of the application 130 into the virtualized computing environment 160, and an application file repository 260 that stores files of various applications registered with the application hosting platform 102. In some embodiments, the application hosting platform 102 may enable an instance of the application 130 in the virtualized computing environment 160 to stream content of the application to a client device 108, such as to an application hosting platform (AHP) GUI described with reference to FIG. 1. In one embodiment, the application 130 may be an example of a gaming application (e.g., a video game) or a collaborative content creation platform or application. In such an instance, the virtualized computing environment 160 may store or load an instance of the gaming application. When a user of the client device initiates a session, the instance of the gaming application may stream game content to the client device 108.

[0025] In at least one embodiment, the application load manager 150 of the application hosting platform 102 can include a scheduler 225. In one embodiment, the scheduler 225 can be configured to initiate or launch an instance of the gaming application 130 for a user of the client device 108. In one embodiment, the scheduler 225 can issue a request to allocate (e.g., create) a virtualized computing environment 160 at the server 106. For example, the scheduler 225 can initiate the creation of the virtualized computing environment 160 by sending a request to a virtual system manager (which manages the creation, teardown, etc. of the virtualized computing environment 160 across the server 106) or by sending a request directly to a hypervisor or host operating system of the server 106. In some embodiments, the scheduler 225 can also send a notification to the software application services 230, which can launch an instance of the gaming application 130 in a preloaded mode into the virtualized computing environment 160.

[0026] In at least one embodiment, the application 130 is integrated with an application hosting platform (AHP) API plugin 235 (e.g., a specified software development kit (SDK)) that can be configured to communicate with the application hosting platform 104 via a predefined set of API commands to launch and / or load an instance of the gaming application 130 on the virtualized computing environment 160. In one embodiment, the AHP API plugin 235 can be configured to determine when the software application 130 is preloaded and fully loaded. In some embodiments, the application hosting platform 130 can have a unique AHP API plugin for each software application 130 that it hosts. Alternatively, a common AHP API plugin can be operational for all software applications hosted / registered on the application hosting platform 102.

[0027] As described with reference to FIG. 1, the software application 130 may include general data and user data (e.g., the application file repository 260 may store the general data and optionally a portion of the user data). As described herein, the application hosting platform may cause the software application 130 to be preloaded before a user of the client device 108 initiates a session. In such an embodiment, the application hosting platform 102 may provide instructions for launching and preloading the software application 130. In at least one embodiment, the application hosting platform 102 may facilitate loading most (or all) of the general data associated with the software application 130 before a user initiates (e.g., launches) the software application 130. This may allow a user to quickly begin streaming content of an instance of the software application 130 to their client device 108, potentially improving the overall user experience.

[0028] For example, the scheduler 225 may issue a request to allocate (or generate) a virtualized computing environment 160 that preloads the gaming application 130. In some embodiments, the scheduler 235 may request to allocate several different virtualized computing environments 160, e.g., so that multiple users who start sessions at different times can access the preloaded instance of the gaming application 130. In one embodiment, the number of virtualized computing environments 160 that the scheduler 225 requests to allocate to a given gaming application 130 may be based on heuristics. For example, in one embodiment, the scheduler 225 may analyze telemetry data (e.g., collected by the gaming application engine 240 or another component) indicative of the average duration that the virtualized computing environments 225 of each software application 130 are idle, or the number of users who are unable to access the preloaded instance of the software application 130. In one embodiment, if the number of users unable to access a preloaded instance of the software application 130 exceeds a threshold number, e.g., one (1), the scheduler 225 may request that additional virtualized computing environments 220 be allocated for each software application 225. For example, if the number of users attempting to run a software application is greater than the number of virtualized computing environments 160 that store preloaded instances of the software application, the application hosting platform 102 may allocate additional virtualized computing environments for the software application. In some embodiments, the threshold number may be based on the popularity of the software application 130 (e.g., how many users request launch).In another embodiment, if the average idle duration exceeds a threshold number, e.g., if a preloaded instance of a software application 130 remains idle for a relatively long period of time (e.g., at least a threshold time period), the scheduler 225 may reduce the number of virtualized computing environments 160 allocated for each software application 130.

[0029] In one embodiment, after the virtualized computing environment 160 is allocated for the preloaded instance of the software application 130, the application hosting platform 102 can communicate with the software application 130 to begin loading the general data. Upon loading the general data, the AHP API plug-in 235 can inform the application hosting platform 102 of the AHP API plug-in that the instance of the software application 130 has been preloaded. In such an embodiment, the virtualized computing environment 160 can remain idle until a user initiates a session. In some embodiments, when a user initiates a session, the application hosting platform 102 can facilitate loading the user data into the virtualized computing environment 220. Thus, the instance of the software application 130 on the virtualized computing environment 160 can be fully loaded and can begin streaming the content (e.g., game or graphics) of the instance of the software application 130 to the user's client device 108.

[0030] In some embodiments, instead of receiving notifications from the AHP API plug-in 235, the application hosting platform 102 may utilize the game application engine 240 to determine when an instance of the game application 130 completes preloading. For example, the game application engine 240 may use a machine learning model trained to predict when an instance of the game application 130 completes preloading. In some embodiments, the machine learning model may be trained using historical data that indicates how long it takes for an application with certain characteristics (e.g., the size of the application file, the type of content created by the application, the graphics and sound effects used, etc.) to preload under certain network conditions. Alternatively or additionally, the machine learning model may be trained specifically for the software application 130. In some cases, the game application engine 240 may determine when the software application 130 completes preloading based on detecting when the software application 130 reaches a main menu screen, e.g., a title sequence. The gaming application engine 240 may collect this data over time indicating how long it took previously created instances of the software application 130 to preload under particular network conditions and provide this collected data as input data to train (or retrain) the machine learning model of the software application 130. In such an embodiment, the gaming application engine 240 may use the trained machine learning model to predict when the software application will be preloaded, for example, to determine the duration before the gaming application is preloaded. If the software application 130 is updated or the duration before launch is changed, the machine learning model can be retrained.Thus, the game application engine 240 can continually improve its accuracy in predicting when a software application will be preloaded. By utilizing the game application engine 240, the application hosting platform 102 can allocate a virtualized computing environment 160 for each software application more efficiently than by waiting for notification from the AHP API plugin 235.

[0031] In one embodiment, several virtualized computing environments 160 for each software application 325 may be preloaded, and a user initiating a session may access the preloaded instances of the software application 130 on a first-in, first-out basis (e.g., first-come, first-served basis). In another embodiment, several virtualized computing environments 160 for each software application 325 may be preloaded, and a user initiating a session may access the preloaded instances of the software application 130 in a tiered system, for example, a premium user may access a preloaded instance of the software application 130 before other users.

[0032] Additional details regarding the process of preloading software application 130 and the set of APIs utilized are described with reference to FIGS.

[0033] FIG. 3 illustrates an example diagram of preloading a software application, according to at least one embodiment. For example, diagram 300 illustrates preloading general data into a software application before a user initiates a session and launches the software application. Diagram 400 can be performed by processing logic including hardware, software, firmware, or any combination thereof. In at least one embodiment, diagram 300 includes operations performed by cloud environment 200 including application hosting platform 102, application 130, application hosting platform (AHP) API plug-in 235, software application services 230, and scheduler 225, as described with reference to FIG. 2. Although shown in a particular sequence or order, the order of processes can be changed unless otherwise specified. Thus, the illustrated embodiment should be understood as merely illustrative, and the illustrated processes can be performed in a different order and some processes can be performed in parallel. Additionally, in various embodiments, one or more processes can be omitted. Thus, not all processes are required in all embodiments. Other diagrams illustrating preloading of a gaming application are possible.

[0034] At operation 305, the software application 130 may begin an initialization procedure using the AHP API plug-in 235. In at least one embodiment, the application hosting platform 102 may integrate the AHP API plug-in (e.g., a specified SDK) 235 into the software application to enable the software application to launch in preloaded mode. That is, the software application may have a startup sequence (e.g., an initialization sequence) and an order in which files are loaded. The application hosting platform 102 may integrate the AHP API plug-in 235 to modify a particular sequence, e.g., send an initialization request to the application hosting platform 102 before the virtualized computing environment 160 is allocated, in order to load as much general data as possible when launching in preloaded mode. In some embodiments, the AHP API plug-in 235 may also be configured to use a particular mechanism to request the software application to launch in preloaded mode. For example, based on information provided by the application developer platform 104 (when registering a software application with the application hosting platform 102), the AHP API plugin 235 can be configured to request that the software application be launched in pre-loaded mode using a CLI command, a flag setting (e.g., in the application's configuration file), or confirmation that the software application exists in the cloud environment.

[0035] At operation 310, the software application 130 may communicate with the AHP API plug-in 235 to verify that the software application will run in a cloud environment. For example, the application developer platform 104 may develop software applications that are played (e.g., loaded locally) in both cloud and non-cloud environments, e.g., an application developer may develop software applications for different contexts. In some embodiments, the software application may include different settings, launch sequences, etc. for each context (e.g., for different computing devices or consoles). Thus, by verifying that the software application will run in a cloud environment, the software application will use the correct settings specified for the cloud environment 200.

[0036] At operation 315, the AHP API plugin 235 may confirm to the software application 130 (e.g., using an API) that the software application is running within a cloud environment. In some embodiments, the AHP API plugin 235 may return a value of “true” when the software application 130 is running within a cloud environment and a value of “false” when the software application 130 is not running within a cloud environment (e.g., when running on a client device). In some embodiments, the AHP API plugin 235 may also communicate to the software application 130 that it should be launched in a preloaded mode when sending the “true value.” For example, at operation 315, the AHP API plugin 235 may issue a command line argument or a flag on disk to initiate the launch of the gaming application based on the initialization at operation 305. In embodiments in which the software application is launched in a preloaded mode based on being in a cloud environment, the software application 130 may initiate the launch before the AHP API plugin 235 communicates the “true” value.

[0037] At operation 320, software application 130 of virtualized computing environment 160 may begin loading general data (e.g., from application file repository 260). In one embodiment, the general data may include assets, textures, artwork for the software application, background graphics, title screen, main menu screen, memory allocation of the software application on the client device, shaders, etc., i.e., all or most of the data excluding user data, for the software application. The amount of general data relative to user data (e.g., the percentage of data that is general data) may be unique for each software application and may be defined by the application developer.

[0038] In operation 325, the software application 130 may send an indication (e.g., via an API) that the preloading is complete. In one embodiment, the software application 130 may issue a message (e.g., RegisterSessionInitCallback or a session initialization callback) via the API indicating that the session can begin, and the software application 130 may enter an idle mode and wait for a callback to resume loading the remaining data, e.g., user data. In some embodiments, the AHP API plug-in 235, the game application engine, or the application hosting platform 102 may predict (e.g., based on machine learning) when the software application preloading is complete, as described with reference to FIG. 2. In such an embodiment, the AHP API plug-in 235 or the application load manager 150 may determine that the software application has completed preloading without a notification API message from the software application 130, and the application hosting platform may forgo operations 325 and 330, for example.

[0039] At operation 330, the AHP API plug-in 235 may notify the software application service 230 that the preloading of the software application is complete. In some embodiments, the software application service 230 may also communicate to the scheduler 225 that the software application has been preloaded. At operation 330, the preloaded instance of the software application may be ready for access by a user.

[0040] At operations 335, 340, and 345, the software application 130 and the AHP API plug-in 235 may be in an idle state. In some embodiments, the virtualized computing environment 160 hosting the pre-loaded instance of the game application may also be in an idle state with respect to the pre-loaded instance of the game application while waiting for a user to initiate a session and continue to load the remainder of the game application.

[0041] At operation 350, the scheduler 225 may indicate to the software application service 230 that a user is initiating a gaming application session. That is, the user may "log in" to their account and identify the gaming application to run, e.g., select the gaming application to play. Additional details regarding user interaction are described with reference to FIGS. 4 and 5.

[0042] In operation 355, the software application service 230 may instruct the AHP API plug-in 335 to initiate a user session for the gaming application.

[0043] At operation 360, the AHP API plug-in 335 may communicate a message (e.g., via an API callback) to the software application 104. In at least one embodiment, the AHP API plug-in 335 may send a message to the software application 130 that should resume operation, e.g., a preloaded software application in the virtualized computing environment 160. In such an embodiment, the AHP API plug-in 235 may communicate a message (e.g., API SessionIntCallback) instructing the software application 130 to resume a previously preloaded software application session. In some embodiments, the AHP API plug-in 235 may also instruct the software application 130 to load remaining data (e.g., user data). Additional details regarding the communication of user data are described with reference to FIGS. 5 and 6.

[0044] At operation 365, the software application may load user data for the software application. In at least one embodiment, the software application may load its title or main menu screen with the user data. That is, the software application 130 may keep data (including user data) loaded until it is ready to be played by the user of the client device 108.

[0045] At operation 370, the software application 130 may communicate its readiness to the AHP API plugin 235. In at least one embodiment, the software application 130 may communicate via an API (e.g., AppReady) to inform the AHP API plugin 235 that the software application has been successfully loaded and is ready to be run by a user.

[0046] In operation 375, the AHP API plug-in 235 may notify the software application service 230 that the software application is loaded and ready for the user to run.

[0047] At operation 380, the software application service 230 may request to begin streaming the graphical or interface content of the instance of the software application. In at least one embodiment, the software content may be streamed to the client device 108. In some embodiments, the client device 108 may be associated with a user who initiated the session. In at least one embodiment, the user may not see the stream until the software application is ready to be interacted with, e.g., until the main menu or title sequence is loaded. In at least one embodiment, the general data is preloaded and the user does not see the game until the software application is ready to be played, thereby reducing wait time for the user and improving the overall user experience.

[0048] FIG. 4 illustrates an example diagram of preloading a software application before a user “logs in” to an account, according to at least one embodiment. For example, diagram 400 illustrates preloading before or while a user begins to access an account associated with the software application. Diagram 400 can be performed by processing logic including hardware, software, firmware, or any combination thereof. In at least one embodiment, diagram 400 includes operations performed by client 108, as described with reference to FIG. 2, and cloud environment 200 including sign-on platform 250 (account manager 205, identity manager 210, account link 215), application hosting platform 102 (software application services 230 and scheduler 225), and software application 130 with AHP API plugin 235 (e.g., an instance of a software application with AHP API plugin on virtualized computing environment 160). Although shown in a particular sequence or order, the order of processes can be changed unless otherwise specified. Thus, the illustrated embodiments should be understood as merely illustrative, and the illustrated processes may be performed in a different order, and some processes may be performed in parallel. Additionally, in various embodiments, one or more processes may be omitted. Thus, not all processes are required in all embodiments. Other diagrams illustrating preloading of software applications are possible.

[0049] At operation 405, the client device 108 and the account linking component 215 may communicate to determine whether an account of a user of the client device 108 is linked to the application hosting platform. In one embodiment, a user may request to launch a software application on the client device 108. In such an embodiment, the client device 108 may check for account linking with the account linking component 215. In one embodiment, the account linking component 215 may determine that there is no account linking. In such an embodiment, the client device 108 may indicate (via a UI) that there is no account linking to the user. In some embodiments, the account linking component 215 may determine that there is an account linking. That is, the account linking component 215 may determine that the user's account is either saved or has been previously used to access the software application. In such an example, the account linking component 215 may proceed to operation 415.

[0050] At operation 410, the account of the user of the client device 108 may be linked to the application hosting platform 102 and the software application 130. In one embodiment, after determining that there is no link, the client device 108 may render a UI that allows the user to enter their credentials and select an option to link their account. In some embodiments, the client device 108 may invoke the account linking via an API communicated to the account manager 205. In such embodiments, the client device 108 may request a redirect universal resource identifier (URI) that allows the user to enter their information. In some embodiments, the account manager 205 may redirect the client device 108 to present the user with a link (e.g., a website link, or a web user interface (UI)) to allow the user to log in to their account associated with the software. In such embodiments, the credentials (e.g., information) are sent to the identity manager 210. In some embodiments, the identity manager 210 may be configured to check or verify the received credentials and information regarding the user's account. For example, the identity manager 210 can validate credentials received from a user via a web UI. In some embodiments, if the account information is correct, the identity manager 210 can generate an access token for the account and generate a refresh token. In some embodiments, if the account information is incorrect, the identity manager 210 can notify the user via the client device 108 that the information is incorrect.

[0051] During operation 410, the identity manager 210 may send a notification to the account linking component that the information provided by the user is correct. In some embodiments, the notification may include an access token and a refresh token.

[0052] At operation 415, the account linking component 215 can store the access token and the refresh token. In at least one embodiment, by storing the access token and the refresh token, the access linking component 215 can link a user account to a software application and an application hosting platform.

[0053] At operation 420, the user may be notified that their account has been linked to the software application and the hosting platform. For example, if the account linking component 215 determines that the account information and / or credentials are stored, the account linking component 215 may return the account name and a token expiration date (e.g., an access token expiration date) to the client device 108. In response to receiving the account name, the client device 108 may cache the account link and the account information. In such an embodiment, the user may access the software application faster because the account information has been previously provided and stored. In response to caching the account information, the client device 108 may present a UI that allows the user to confirm that the account has been successfully linked.

[0054] In some embodiments, operations 425-445 may be performed before or at the same time that a user links their account to the software application and the application hosting platform. That is, the application hosting platform may enable preloading of an instance of the software application (as described with reference to FIGS. 2 and 3) before a user begins to initiate a session. As described with reference to FIGS. 2 and 3, a scheduler may request to allocate a preload for an instance of the software application. In at least one embodiment, the request to allocate a preload may be based on the heuristic problem solving method described with reference to FIG. 2. In some embodiments, the application hosting platform may request to allocate several preloads, e.g., a single software application may be preloaded several times across multiple virtualized computing environments 160.

[0055] In operation 430, the software application service 230 may launch the software application in a preloaded mode via the AHP API plugin 235. As described with reference to FIGURE 4, the software application may be launched by the AHP API plugin 235 using a CLI command, a flag setting (in the application's configuration file), or confirmation that the software application is running in a cloud environment.

[0056] In operation 435, the virtualized computing environment 160 can be pre-loaded with all (or most) of the software application's general data, for example, an application developer platform can load the general data into an instance of the software application in the virtualized computing environment 160.

[0057] At operation 440, an instance of the software application on the hosting platform may be idled. In at least one embodiment, the instance of the software application may be idled and wait for a user to initiate a session. Because the instance of the software application is preloaded with general data, load times are reduced after a user initiates a session, improving the overall user experience.

[0058] In operation 445, the software application may send (e.g., via an API message) an indication that the software application is preloaded and waiting for a callback when the user initiates a session, as described with reference to FIG.

[0059] FIG. 5 illustrates an example diagram of preloading software applications and loading user data, according to at least one embodiment. For example, diagram 500 illustrates preloading before a user initiates a session and loading user data thereafter. Diagram 500 can be performed by processing logic including hardware, software, firmware, or any combination thereof. In at least one embodiment, diagram 500 includes operations performed by client 108, as described with reference to FIG. 2, and cloud environment 200 including sign-on platform 250 (account manager 205, identity manager 210, account link 215), application hosting platform 102 (software application services 230 and scheduler 225), and software application 130 having AHP API plugin 235 (e.g., an instance of a software application having AHP API plugin on virtualized computing environment 160). Although shown in a particular sequence or order, the order of processes can be changed unless otherwise specified. Thus, the illustrated embodiments should be understood as merely illustrative, and the illustrated processes may be performed in a different order, and some processes may be performed in parallel. Additionally, in various embodiments, one or more processes may be omitted. Thus, not all processes are required in all embodiments. Other diagrams illustrating preloading of software applications are possible.

[0060] In operation 505, the application hosting platform 102 may launch an instance of the software application in a preloaded mode, as described with reference to operations 425-445 of FIG. 4. That is, the scheduler 225 of the application hosting platform 102 may allocate a virtualized computing environment for the instance of the software application 130 (e.g., using a running virtual machine (VM) or adding a new VM) and instruct the instance of the gaming application 130 to be launched and preloaded by loading general data. The software application 130 may load the general data, register a session callback with the AHP API plug-in 235, and then go into an idle state and wait for an API session callback from the AHP API plug-in 235. In some embodiments, the virtualized computing environment 160 may notify the scheduler 225 that the virtualized computing environment 160 is available for a user session and transition to an idle state.

[0061] At operation 510, the user may initiate a software application 130 session and have his / her account information verified for launch of the software application. For example, the user may initiate the launch of the software application on the client device 108 and may request, e.g., via a UI, to launch the software application. In response to initiating the launch, the client device 108 may present the user with a loading UI, e.g., a loading screen, a notification that the software application has started to load, and the like. Additionally, in some embodiments, the client device 108 may check a cache (e.g., a local cache) to see if the user account is remembered based on a previous account link, as described with reference to FIG. 4. In some embodiments, if the client device 108 cannot find the account in the cache, the user may be prompted to enter his / her credentials, as described with reference to operation 410 of FIG. 4. In at least one embodiment, the client device 108 may find the account information in the cache. In such an embodiment, the client device 108 may communicate with the scheduler 225 of the application hosting platform 102. For example, the client device 108 may request to launch an instance of the software application with a user account, e.g., via an account-linked flag or an instruction to load information associated with the user account into the software application. In response to receiving the flag, the scheduler 225 may request an exchange code from the account linking component to receive information associated with the user account, e.g., user data.

[0062] During operation 510, in some embodiments, the account linking component 215 can redeem the stored access and refresh tokens with the identity manager 210 to access information associated with the user account. That is, as described with reference to operation 415 of FIG. 4, the account linking component 215 can store the access and refresh tokens when the user “logs in” to their account. In some embodiments, the account linking component 215 can access the access and refresh tokens to re-access the account information. In some embodiments, in response to receiving the access and refresh tokens, the identity manager 210 can validate the access and refresh tokens, generate an exchange code, and generate new access and refresh tokens. That is, the identity manager 210 can validate that the tokens received from the account linking component 215 are associated with the user account, e.g., the information and credentials are correct. If the identity manager 210 verifies the token from the account linking component 215, the identity manager 210 can send the generated exchange code and new access and refresh tokens to the account linking component. In some embodiments, the account linking component can store the new access and refresh tokens and send the exchange code to the scheduler 225. Thus, the application hosting platform 102 can validate the user account and access user data associated with each launched software application.

[0063] At operation 515, the scheduler 225 may request to initiate a session with a software application for the user. In some embodiments, the scheduler 225 may send a notification to the gaming application service 230 to initiate a session with information associated with the user, e.g., user data.

[0064] At operation 520, the software application service may notify the AHP API plug-in 235 to communicate with the software application 130 to initiate a session for the user and load the user data. In some embodiments, the software application service maintains a list of pre-loaded instances of the software application 130 on one or more virtualized computing environments 160, selects one of the pre-loaded instances from the list, identifies a corresponding AHP API plug-in 235 on the respective virtualized computing environment 160, and notifies it to communicate with the associated instance of the software application 130.

[0065] At operation 525, the AHP API plug-in 235 may communicate with the software application 130 to initiate a session of the software application using the preloaded instance of the software application. For example, the AHP API plug-in 235 may communicate an API session callback (e.g., SessionInitCallback) to the software application. In some embodiments, the AHP API plug-in 235 may communicate the session callback to initiate loading of user data into the software application instance preloaded at operation 505 as described above. In some embodiments, the AHP API plug-in 235 may instruct the software application to load the exchange code received from the software application service into the software application, e.g., to load the software application with the user data.

[0066] At operation 530, the software application can send an exchange code to the identity manager to log into the user's account. In some embodiments, the software application can access the user's account to retrieve user data, such as specific user settings (e.g., resolution or shader settings), custom user content (e.g., customizations for an in-game character created by the user), custom key bindings, etc. In some embodiments, the software application can communicate with the identity manager 210 to validate the credentials and access user information.

[0067] In operation 535, the identity manager 210 may verify the exchange code. In at least one embodiment, the identity manager 210 associated with the software application may utilize the exchange code to initiate a session between the software application and the user data. For example, if the identity manager 210 verifies the exchange code (e.g., if the exchange code matches a previously generated exchange code), the identity manager 210 may allow the software application to log into the user account and access the information.

[0068] In operation 540, the identity manager 210 may indicate to the software application that the exchange code is valid and login to the user account is successful.

[0069] At operation 545, the software application may log into the user account and begin loading user data. In at least one embodiment, an instance of the software application in the virtualized computing environment 160 may be loaded with specific user settings (e.g., resolution or shader settings), custom user content (e.g., customizations for an in-game character created by a user), custom key bindings, etc. based on the login to the user account, e.g., allowing the software application to customize the user experience based on the user's preferences and user account settings.

[0070] At operation 550, the software application may communicate with AHP API plugin 235 via an API (e.g., calling AppReady). In some embodiments, the software application may notify AHP API plugin 235 that an instance of the software application in virtualized computing environment 160 is ready for a user. That is, the software application may notify AHP API plugin 235 that both user data and general data have been loaded and that the user may access the main menu or title sequence of the software application.

[0071] In operation 555, the AHP API plug-in 235 may notify the software application service 230 that the instance of the software application is loaded and ready for the user.

[0072] At operation 560, the software application service 230 may begin streaming the gaming or graphics content of the instance of the software application to the client device 108.

[0073] At operation 565, the client device 108 (e.g., a client component of the software application or a web browser) may display the streamed content of the instance of the software application to the user via a UI.

[0074] At operation 570, the user may begin playing the game, e.g., interact with characters, progress further in the storyline, play in multiplayer mode, etc. In some embodiments, by pre-loading much of the general data before the user starts or launches the software application, user wait times are reduced and the overall user experience is improved.

[0075] FIG. 6 illustrates a flow diagram of a method 600 for preloading software applications in a cloud computing environment. Method 600 may be performed by processing logic including hardware, software, firmware, or any combination thereof. In at least one embodiment, method 600 is performed by computing environment 200 as described with reference to FIGS. 2-5. Although shown in a particular sequence or order, the order of processes may be changed unless otherwise specified. Thus, the illustrated embodiment should be understood as merely illustrative, and the illustrated processes may be performed in a different order and some processes may be performed in parallel. Furthermore, in various embodiments, one or more processes may be omitted. Thus, not all processes are required in all embodiments. Other diagrams illustrating methods for preloading software applications are possible.

[0076] At operation 605, the processing logic may send a preload request using the application hosting platform to preload a first portion of data for an application hosted using the application hosting platform. In one embodiment, the processing logic may determine to allocate a number of virtualized computing environments to launch an instance of the application in a preloaded mode. In some embodiments, the processing logic may determine the number of virtualized computing environments to preload the application on based on an average duration of other virtualized computing environments in which the preloaded portion is idle. For example, if the idle time is relatively long (e.g., below a threshold indicating that the user is not running the application), the processing logic may determine to allocate fewer virtualized computing environments for the application. In some embodiments, the preload request may be received before user input identifying an application to run is received. That is, the processing logic may request the preload before the user selects a game to run. In some embodiments, the first portion of data may be general data, e.g., data excluding data related to the user request. In some embodiments, the processing logic may include a command line interface or an on-disk flag in the first request to load the first portion of data. In some embodiments, the processing logic may send multiple preload requests using an application hosting platform to preload a first portion of the data to multiple virtualized computing environments hosted by one or more computer systems in a cloud platform.In at least one embodiment, the application may include at least one of a gaming application, a video conferencing application, a content creation application, a cloud-hosted application, a collaborative content creation application, a cloud-hosted collaborative content creation application, a video streaming application, a machine learning application, or a simulation application. In some embodiments, the application hosting platform may include at least one of a cloud-hosted gaming services platform, a cloud-hosted collaborative content creation platform for disparate content creation applications, a video streaming hosting platform, a testing platform for simulated or augmented content, a machine learning training platform, a machine learning deployment platform, or a video conferencing hosting platform.

[0077] At operation 610, the processing logic may receive a first indication using the application hosting platform that a first portion of data has been preloaded for an application using the virtualized computing environment. In some embodiments, the virtualized computing environment may be transitioned to an idle mode upon receiving the indication. That is, when the preloading has occurred and no user request is received, e.g., when the application has not yet been fully loaded, the processing logic may place the virtualized computing environment in an idle mode. In some embodiments, the processing logic may predict when a first portion of data will be preloaded for an application using the virtualized computing environment. In such embodiments, the processing logic may monitor and store and / or access an average duration it takes to preload an instance of the application in the virtualized computing environment and use this data to train a machine learning model. The processing logic may utilize the trained machine learning training model to predict when the preloading will occur. In some embodiments, the processing logic may receive a third indication that a first portion of data has been preloaded to a plurality of virtualized computing environments.

[0078] At operation 615, the processing logic may receive a user request to run an application using the application hosting platform. In one embodiment, the processing logic may receive the user request upon receiving the first indication. In some embodiments, the user request may be associated with a user at a client device. In some embodiments, the processing logic may receive an indication from the application of an update to one or more files of the application prior to receiving the user request. In such embodiments, the processing logic may tear down the virtualized computing environment and preload a second virtualized computing environment with the updated files in response to receiving the indication that one or more files associated with the application have been updated. In some embodiments, the processing logic may receive a second user request to run an application. In such embodiments, the processing logic may execute an application that runs in a second virtualized computing environment from the plurality of virtualized computing environments.

[0079] At operation 620, the processing logic may send a load request to load a second portion of the data for the application to the virtualized computing environment using the application hosting platform. In one embodiment, the second portion of the data may be associated with user data, e.g., data associated with a user input or a user request. In at least one embodiment, the virtualized computing environment may transition from an idle state to an active state to load the second portion of the data. In some embodiments, the processing logic may also send a second load request to load the second portion of the data for the application in the second virtualized computing environment.

[0080] At operation 625, the processing logic may receive a second indication that a second portion of data has been loaded into the virtualized computing environment for the application using the application hosting platform. In one embodiment, the processing logic may receive an indication that user data has been loaded for the application in the virtualized computing environment. In some embodiments, the second indication may also inform the processing logic that the application in the virtualized computing environment is ready to be launched, e.g., ready for a user to engage. In some embodiments, the processing logic may receive a fourth indication that a second portion of data has been loaded for the application in the second virtualized computing environment.

[0081] At operation 630, the processing logic may cause the application to run using the virtualized computing environment such that content of the instance of the application may be streamed to a user's device, e.g., a user may initiate playback of the application. In some embodiments, the processing logic may cause the application to run in a second virtualized computing environment in response to a second user request.

[0082] Figure 7A illustrates inference and / or training logic 715 that may be used to perform inference and / or training operations associated with one or more embodiments. More details regarding inference and / or training logic 715 are provided below in conjunction with Figures 7A and / or 7B.

[0083] In at least one embodiment, the inference and / or training logic 715 may include, without limitation, code and / or data storage 701 for storing forward and / or output weights, and / or input / output data, and / or other parameters for configuring neurons or layers of a neural network that is trained and / or used to infer in one or more embodiments. In at least one embodiment, the training logic 715 may include or be coupled to code and / or data storage 701 for storing graph code or other software for timing and / or ordering control into which the weights and / or other parameter information is loaded to configure logic including integer and / or floating point units (collectively arithmetic logic units (ALUs)). In at least one embodiment, code such as graph code may be configured to transmit the weights or other parameter information to a processor ALU based on the architecture of the neural network to which the code corresponds. LU. In at least one embodiment, code and / or data storage 701 stores weight parameters and / or input / output data for each layer of a neural network trained or used in conjunction with one or more embodiments during forward propagation of the input / output data and / or weight parameters during training and / or inference using aspects of one or more embodiments. In at least one embodiment, any portion of code and / or data storage 701 may be included with other on-chip or off-chip data storage, including L1, L2, or L3 caches of a processor, or system memory.

[0084] In at least one embodiment, any portion of the code and / or data storage 701 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or code and / or data storage 701 may be cache memory, dynamic randomly addressable memory (“DRAM”), static randomly addressable memory (“SRAM”), non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the choice of whether the code and / or code and / or data storage 701 is internal or external to the processor, for example, or comprised of DRAM, SRAM, flash, or some other type of storage, may depend on the storage available on-chip versus off-chip, the latency requirements of the training and / or inference functions being performed, the batch size of data used in the inference and / or training of the neural network, or any combination of these factors.

[0085] In at least one embodiment, the inference and / or training logic 715 may include, without limitation, code and / or data storage 705 for storing backpropagation and / or output weights and / or input / output data corresponding to neurons or layers of a neural network trained and / or used to infer with one or more aspects of the embodiment. In at least one embodiment, the code and / or data storage 705 stores weight parameters and / or input / output data for each layer of a neural network trained or used in conjunction with one or more embodiments while backpropagating input / output data and / or weight parameters during training and / or inference using aspects of one or more embodiments. In at least one embodiment, the training logic 715 may include or be coupled to code and / or data storage 705 for storing graph code or other software for controlling the timing and / or sequence, where the weights and / or other parameter information are loaded into logic including integer and / or floating point units (collectively arithmetic logic units (ALUs). In at least one embodiment, code such as the graph code loads the weights or other parameter information into the processor ALUs based on the architecture of the neural network to which the code corresponds. In at least one embodiment, the training logic 715 may include code and / or data storage 705 for storing the graph code or other software for controlling the timing and / or sequence, where the weights and / or other parameter information are loaded into the processor ALUs. In an embodiment, any portion of code and / or data storage 705 may be included with other on-chip or off-chip data storage, including L1, L2, or L3 cache of a processor, or system memory. In at least one embodiment, any portion of code and / or data storage 705 may be internal or external to one or more processors, or other hardware logic devices or circuits. In at least one embodiment, code and / or data storage 705 may be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage.In at least one embodiment, the choice of whether the code and / or data storage 705 is internal or external to the processor, for example, or comprised of DRAM, SRAM, flash, or some other type of storage, may depend on the storage available on-chip versus off-chip, the latency requirements of the training and / or inference functions being performed, the batch size of data used in inference and / or training of the neural network, or any combination of these factors.

[0086] In at least one embodiment, code and / or data storage 701 and code and / or data storage 705 may be separate storage structures. In at least one embodiment, code and / or data storage 701 and code and / or data storage 705 may be the same storage structure. In at least one embodiment, code and / or data storage 701 and code and / or data storage 705 may be partially the same storage structure and partially separate storage structures. In at least one embodiment, any portion of code and / or data storage 701 and code and / or data storage 705 may be included with other on-chip or off-chip data storage, including a processor's L1, L2, or L3 cache, or system memory.

[0087] In at least one embodiment, the inference and / or training logic 715 may include one or more arithmetic logic units ("ALUs") 710, including without limitation integer and / or floating point units, for performing logical and / or arithmetic operations based at least in part on or indicated by the training and / or inference code (e.g., graph code), the results of which may generate activations (e.g., output values ​​from layers or neurons in a neural network) that are stored in activation storage 720 and that are functions of input / output and / or weight parameter data stored in code and / or data storage 701 and / or code and / or data storage 705. In at least one embodiment, the activations stored in activation storage 720 are generated according to linear algebra and / or matrix based calculations performed by ALU 710 in response to executing instructions or other code, where weight values ​​stored in code and / or data storage 705 and / or code and / or data storage 701 are used as operands along with other values, such as bias values, gradient information, momentum values, or other parameters or hyperparameters, any or all of which may be stored in code and / or data storage 705, or code and / or data storage 701, or in another storage, on-chip or off-chip.

[0088] In at least one embodiment, ALU 710 is included within one or more processors or other hardware logic devices or circuits, while in other embodiments ALU 710 may be external to the processors or other hardware logic devices or circuits that use them (e.g., a coprocessor). In at least one embodiment, ALU 710 may be included within an execution unit of a processor, or may be otherwise included within an ALU bank accessible by execution units of a processor, either within the same processor or distributed among different processors of different types (e.g., central processing unit, graphics processing unit, fixed function unit, etc.). In at least one embodiment, code and / or data storage 701, code and / or data storage 705, and activation storage 720 may be in the same processor or other hardware logic devices or circuits, while in other embodiments they may be in different processors or other hardware logic devices or circuits, or some combination of the same processor or other hardware logic devices or circuits and different processors or other hardware logic devices or circuits. In at least one embodiment, any portion of activation storage 720 may be included with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache, or system memory. Additionally, inference and / or training code may be stored with other code accessible to the processor or other hardware logic or circuitry, and may be fetched and / or processed using the processor's fetch, decode, schedule, execute, retire, and / or other logic circuitry.

[0089] In at least one embodiment, activation storage 720 may be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, activation storage 720 may be fully or partially internal or external to one or more processors or other logic circuitry. In at least one embodiment, the choice of whether activation storage 720 is internal or external to a processor, for example, or comprised of DRAM, SRAM, flash, or some other type of storage, may depend on the storage available on-chip versus off-chip, the latency requirements of the training and / or inference functions being performed, the batch size of data used in inference and / or training of the neural network, or any combination of these factors. In at least one embodiment, the inference and / or training logic 715 shown in Figure 7A may be used in conjunction with an application specific integrated circuit ("ASIC"), such as a Tensorflow® processing unit from Google, an inference processing unit (IPU) from Graphcore™, or a Nervana® (e.g., "Lake Crest") processor from Intel Corporation. In at least one embodiment, the inference and / or training logic 715 shown in Figure 7A may be used in conjunction with other hardware, such as central processing unit ("CPU") hardware, graphics processing unit ("GPU") hardware, or data processing unit ("DPU") hardware, or a field programmable gate array ("FPGA").

[0090] FIG. 7B illustrates inference and / or training logic 715, according to at least one or more embodiments. In at least one embodiment, the inference and / or training logic 715 may include, without limitation, hardware logic in which computational resources are dedicated to or otherwise used only in conjunction with weight values ​​or other information corresponding to one or more layers of neurons in a neural network. In at least one embodiment, the inference and / or training logic 715 illustrated in FIG. 7B may be used in conjunction with an application specific integrated circuit (ASIC), such as a Tensorflow® processing unit from Google, an inference processing unit (IPU) from Graphcore™, or a Nervana® (e.g., “Lake Crest”) processor from Intel Corporation. In at least one embodiment, the inference and / or training logic 715 illustrated in FIG. 7B may be used in conjunction with other hardware, such as central processing unit (CPU) hardware, graphics processing unit (GPU) hardware, or data processing unit ("DPU") hardware, or a field programmable gate array (FPGA). In at least one embodiment, the inference and / or training logic 715 may include, without limitation, code and / or data storage 701 and code and / or data storage 705, which may be used to store code (e.g., graph code), weight values, and / or other information including bias values, gradient information, momentum values, and / or other parameter or hyperparameter information. In at least one embodiment shown in FIG. 7B, each of the code and / or data storage 701 and code and / or data storage 705 is associated with dedicated computational resources, such as computation hardware 702 and computation hardware 706, respectively. In at least one embodiment, each of the computation hardware 702 and computation hardware 706 includes one or more ALUs that perform mathematical functions, such as linear algebraic functions, only on the information stored in the code and / or data storage 701 and code and / or data storage 705, respectively, with the results stored in activation storage 720.

[0091] In at least one embodiment, each of the code and / or data storage 701 and 705 and corresponding computational hardware 702 and 706 correspond to a different layer of a neural network, such that activations resulting from one “storage / computation pair 701 / 702” of the code and / or data storage 701 and computational hardware 702 are provided as inputs to a “storage / computation pair 705 / 706” of the code and / or data storage 705 and computational hardware 706 to reflect the conceptual organization of the neural network. In at least one embodiment, each of the storage / computation pairs 701 / 702 and 705 / 706 may correspond to two or more layers of the neural network. In at least one embodiment, additional storage / computation pairs (not shown) may be included in the inference and / or training logic 715 after or in parallel with the storage / computation pairs 701 / 702 and 705 / 706.

[0092] 8 illustrates an example data center 800 in which at least one embodiment may be used. In at least one embodiment, the data center 800 includes a data center infrastructure layer 810, a framework layer 820, a software layer 830, and an application layer 840.

[0093] 8, in at least one embodiment, data center infrastructure layer 810 may include a resource orchestrator 812, grouped computing resources 814, and node computing resources ("node CRs") 816(1)-816(N), where "N" represents any whole positive integer. In at least one embodiment, node CRs 816(1)-816(N) may include, but are not limited to, any number of central processing units ("CPUs") or other processors (including accelerators, field programmable gate arrays (FPGAs), data processing units, graphics processors, etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid state drives or disk drives), network input / output ("NW I / O") devices, network switches, virtual machines ("VMs"), power modules, and cooling modules. In at least one embodiment, one or more of the nodes CR 816(1)-816(N) may be a server having one or more of the computing resources described above.

[0094] In at least one embodiment, the grouped computing resources 814 may include separate groups of node CRs housed in one or more racks (not shown), or multiple racks housed in a data center at various graphical locations (also not shown). The separate groups of node CRs in the grouped computing resources 814 may include grouped compute resources, network resources, memory resources, or storage resources that may be configured or allocated to support one or more workloads. In at least one embodiment, several node CRs including CPUs or processors may be grouped in one or more racks to provide compute resources to support one or more workloads. In at least one embodiment, the one or more racks may also include any number of power modules, cooling modules, and network switches in any combination.

[0095] In at least one embodiment, resource orchestrator 812 may configure or otherwise control one or more nodes CR 816(1)-816(N) and / or grouped computing resources 814. In at least one embodiment, resource orchestrator 812 may include a software design infrastructure ("SDI") management entity for data center 800. In at least one embodiment, resource orchestrator may include hardware, software, or some combination thereof.

[0096] As shown in FIG. 8 , in at least one embodiment, framework layer 820 includes a job scheduler 822, a configuration manager 824, a resource manager 826, and a distributed file system 828. In at least one embodiment, framework layer 820 may include a framework for supporting software 832 in software layer 830 and / or one or more applications 842 in application layer 840. In at least one embodiment, software 832 or applications 842 may each include web-based service software or applications, such as those offered by Amazon Web Services, Google Cloud, and Microsoft Azure. In at least one embodiment, framework layer 820 may be a type of free and open source software web application framework, such as, but not limited to, Apache Spark™ (hereinafter “Spark”), which can use distributed file system 828 for large-scale data processing (e.g., “big data”). In at least one embodiment, the job scheduler 822 may include a Spark driver to facilitate scheduling of workloads supported by various tiers of the data center 800. In at least one embodiment, the configuration manager 824 may be capable of configuring different tiers, such as the software tier 830, as well as the framework tier 820 including Spark and a distributed file system 828 to support large scale data processing. In at least one embodiment, the resource manager 826 may be capable of managing clustered or grouped computing resources that are mapped or allocated to support the distributed file system 828 and the job scheduler 822. In at least one embodiment, the clustered or grouped computing resources may include grouped computing resources 814 in the data center infrastructure tier 810.In at least one embodiment, resource manager 826 may cooperate with resource orchestrator 812 to manage these mappings or allocated computing resources.

[0097] In at least one embodiment, software 832 included in software layer 830 may include software used by at least a portion of nodes CR 816(1)-816(N), grouped computing resources 814, and / or distributed file system 828 of framework layer 820. The one or more types of software may include, but are not limited to, Internet web page searching software, email virus scanning software, database software, and streaming video content software.

[0098] In at least one embodiment, the applications 842 included in the application layer 840 may include one or more types of applications used by at least a portion of the nodes CRs 816(1)-816(N), the grouped computing resources 814, and / or the distributed file system 828 of the framework layer 820. The one or more types of applications may include, but are not limited to, any number of genomics applications, cognitive compute, and machine learning applications including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), or other machine learning applications used in conjunction with one or more embodiments.

[0099] In at least one embodiment, any of configuration manager 824, resource manager 826, and resource orchestrator 812 may implement any number and type of self-correcting actions based on any amount and type of data obtained in any technically feasible manner. In at least one embodiment, the self-correcting actions may enable data center operators of data center 800 to avoid determining potentially poor configurations and eliminate underutilized and / or underperforming portions of the data center.

[0100] In at least one embodiment, data center 800 may include tools, services, software, or other resources for training one or more machine learning models or predicting or inferring information using one or more machine learning models according to one or more embodiments described herein. For example, in at least one embodiment, a machine learning model may be trained by calculating weight parameters according to a neural network architecture using the software and computing resources described above with respect to data center 800. In at least one embodiment, a trained machine learning model corresponding to one or more neural networks may be used to infer or predict information using the resources described above with respect to data center 800 by using weight parameters calculated by one or more techniques described herein.

[0101] In at least one embodiment, the data center may use a CPU, application specific integrated circuit (ASIC), GPU, DPU, FPGA, or other hardware to perform training and / or inference using the resources described above. Additionally, one or more of the software and / or hardware resources described above may be configured as a service to enable a user to train or perform inference on information, such as image recognition, speech recognition, or other artificial intelligence services.

[0102] Inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. More details regarding inference and / or training logic 715 are provided below in conjunction with Figures 7A and / or 7B. In at least one embodiment, inference and / or training logic 715 may be used in the system of Figure 8 for inference or prediction operations based at least in part on weight parameters calculated using neural network training operations, neural network functionality and / or architecture, or neural network use cases described herein.

[0103] These components can be used to generate synthetic data that mimics failure cases in the network training process, which may help improve the network's performance while limiting the amount of synthetic data to avoid overfitting.

[0104] 9 is a block diagram illustrating an exemplary computer system, which may be a system having interconnected devices and components formed with a processor that may include an execution unit for executing instructions, a system on a chip (SoC), or some combination of these 900, according to at least one embodiment. In at least one embodiment, computer system 900 may include components such as, without limitation, a processor 902 for using an execution unit that includes logic for executing algorithms for processing data in accordance with the present disclosure, such as in the embodiments described herein. In at least one embodiment, computer system 900 may include a processor such as the PENTIUM® processor family, Xeon™, Itanium®, XScale™, and / or StrongARM™, Intel® Core™, or Intel® Nervana™ microprocessors available from Intel Corporation, Santa Clara, Calif., although other systems may be used (including PCs with other microprocessors, engineering workstations, set-top boxes, etc.). In at least one embodiment, computer system 900 may run a version of the WINDOWS operating system available from Microsoft Corporation of Redmond, Washington, although other operating systems (e.g., UNIX and Linux), embedded software, and / or graphical user interfaces may be used.

[0105] Embodiments may be used in other devices, such as portable devices and embedded applications. Some examples of portable devices include cellular phones, Internet Protocol devices, digital cameras, personal digital assistants ("PDAs"), and portable PCs. In at least one embodiment, embedded applications may include microcontrollers, digital signal processors ("DSPs"), systems-on-chips, network computers ("NetPCs"), set-top boxes, network hubs, wide area network ("WAN") switches, edge devices, Internet-of-Things ("IoT") devices, or any other system capable of executing one or more instructions according to at least one embodiment.

[0106] In at least one embodiment, computer system 900 may include, without limitation, a processor 902, which may include one or more execution units 908 for performing training and / or inference of machine learning models according to the techniques described herein. In at least one embodiment, computer system 900 is a single processor desktop or server system, while in other embodiments, computer system 900 may be a multiprocessor system. In at least one embodiment, processor 902 may include, without limitation, a complex instruction set computer ("CISC") microprocessor, a reduced instruction set computing ("RISC") microprocessor, a very long instruction word ("VLIW") microprocessor, a processor implementing a combination of instruction sets, or any other processor device, such as a digital signal processor. In at least one embodiment, processor 902 may be coupled to a processor bus 910, which may transmit data signals between processor 902 and other components within computer system 900.

[0107] In at least one embodiment, the processor 902 may include, without limitation, a level 1 ("L1") internal cache memory ("cache") 904. In at least one embodiment, the processor 902 may have a single internal cache or multiple levels of internal cache. In at least one embodiment, the cache memory may be external to the processor 902. Other embodiments may include a combination of both internal and external cache depending on the particular implementation and needs. In at least one embodiment, the register file 906 may store different types of data in various registers, including, without limitation, integer registers, floating point registers, status registers, and an instruction pointer register.

[0108] In at least one embodiment, processor 902 also includes an execution unit 908, including, without limitation, logic for performing integer and floating point operations. In at least one embodiment, processor 902 may also include a microcode ("u-code") read only memory ("ROM") that stores microcode for certain macroinstructions. In at least one embodiment, execution unit 908 may include logic for a packed instruction set 909. In at least one embodiment, by including packed instruction set 909 in the instruction set of general purpose processor 902 along with associated circuitry for executing the instructions, operations used by many multimedia applications may be performed using packed data in general purpose processor 902. In one or more embodiments, many multimedia applications may be accelerated and executed more efficiently by performing operations on packed data using the full width of the processor's data bus, thereby eliminating the need to transfer smaller units of data between the processor's data bus to perform one or more operations on one data element at a time.

[0109] In at least one embodiment, the execution unit 908 may also be used in a microcontroller, an embedded processor, a graphics device, a DSP, and other types of logic circuits. In at least one embodiment, the computer system 900 may include, without limitation, a memory 920. In at least one embodiment, the memory 920 may be implemented as a dynamic random access memory ("DRAM") device, a static random access memory ("SRAM") device, a flash memory device, or other memory device. In at least one embodiment, the memory 920 may store instructions 919 and / or data 921 represented by data signals that may be executed by the processor 902.

[0110] In at least one embodiment, a system logic chip may be coupled to the processor bus 910 and memory 920. In at least one embodiment, the system logic chip may include, without limitation, a memory controller hub ("MCH") 916, and the processor 902 may communicate with the MCH 916 via the processor bus 910. In at least one embodiment, the MCH 916 may provide a high bandwidth memory path 918 to the memory 920 for storing instructions and data, and for storing graphics commands, data, and textures. In at least one embodiment, the MCH 916 may route data signals between the processor 902, the memory 920, and other components of the computer system 900, and may bridge data signals between the processor bus 910, the memory 920, and a system I / O interface 922. In at least one embodiment, the system logic chip may provide a graphics port for coupling to a graphics controller. In at least one embodiment, the MCH 916 may be coupled to memory 920 via a high bandwidth memory path 918, and the graphics / video card 912 may be coupled to the MCH 916 via an Accelerated Graphics Port ("AGP") interconnect 914.

[0111] In at least one embodiment, computer system 900 may use system I / O 922, a proprietary hub interface bus, to couple MCH 916 to an I / O controller hub ("ICH") 930. In at least one embodiment, ICH 930 may provide direct connections to some I / O devices via a local I / O bus. In at least one embodiment, the local I / O bus may include, without limitation, a high-speed I / O bus for connecting peripherals to memory 920, a chipset, and processor 902. Examples may include, without limitation, an audio controller 929, a firmware hub ("Flash BIOS") 928, a wireless transceiver 926, data storage 924, a legacy I / O controller 923 including a user input and keyboard interface 925, a serial expansion port 927 such as a Universal Serial Bus ("USB"), and a network controller 934, which may in some embodiments comprise a data processing device. Data storage 924 may comprise a hard disk drive, a floppy disk drive, a CD-ROM device, a flash memory device, or other mass storage device.

[0112] In at least one embodiment, Figure 9 illustrates a system including interconnected hardware devices or "chips," while in other embodiments, Figure 9 may illustrate an exemplary system-on-a-chip ("SoC"). In at least one embodiment, the devices may be interconnected with proprietary interconnects, standard interconnects (e.g., PCIe), or some combination thereof. In at least one embodiment, one or more components of computer system 900 may be interconnected using a compute express link (CXL) interconnect.

[0113] Inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. More details regarding inference and / or training logic 715 are provided below in conjunction with Figures 7A and / or 7B. In at least one embodiment, inference and / or training logic 715 may be used in the system of Figure 9 for inference or prediction operations based at least in part on weight parameters calculated using neural network training operations, neural network functionality and / or architecture, or neural network use cases described herein.

[0114] These components can be used to generate synthetic data that mimics failure cases in the network training process, which may help improve the performance of the network while limiting the amount of synthetic data to avoid overfitting.

[0115] 10 is a block diagram illustrating an electronic device 1000 for utilizing a processor 1010, according to at least one embodiment. In at least one embodiment, the electronic device 1000 may be, for example, without limitation, a notebook, a tower server, a rack server, a blade server, a laptop, a desktop, a tablet, a mobile device, a phone, an embedded computer, an edge device, an IoT device, or any other suitable electronic device.

[0116] In at least one embodiment, system 1000 may include a processor 1010 communicatively coupled to any suitable number or type of components, peripherals, modules, or devices, without limitation. In at least one embodiment, processor 1010 is coupled using a bus or interface, such as a 1°C bus, a System Management Bus (“SMBus”), a Low Pin Count (LPC) bus, a Serial Peripheral Interface (“SPI”), a High Definition Audio (“HDA”) bus, a Serial Advance Technology Attachment (“SATA”) bus, a Universal Serial Bus (“USB”) (versions 1, 2, 3), or a Universal Asynchronous Receiver / Transmitter (“UART”) bus. In at least one embodiment, FIG. 10 illustrates a system including interconnected hardware devices or "chips," while in other embodiments, FIG. 10 may illustrate an exemplary system-on-a-chip ("SoC"). In at least one embodiment, the devices illustrated in FIG. 10 may be interconnected with proprietary interconnects, standard interconnects (e.g., PCIe), or some combination thereof. In at least one embodiment, one or more components of FIG. 10 are interconnected using a Compute Express Link (CXL) interconnect.

[0117] In at least one embodiment, FIG. 10 illustrates a display 1024, a touch screen 1025, a touch pad 1030, a Near Field Communications unit ("NFC") 1045, a sensor hub 1040, a thermal sensor 1046, an Express Chipset ("EC") 1035, a Trusted Platform Module ("TPM") 1038, a BIOS / firmware / flash memory ("BIOS,FW flash") 1022, a DSP 1060, a drive 1020, such as a Solid State Disk ("SSD") or a Hard Disk Drive ("HDD"), a wireless local area network unit ("WLAN") 1050, a Bluetooth unit 1052, a Wireless Wide Area Network unit ("WWAN") 1054, a Bluetooth 1056, a Bluetooth 2000 (BLE) 1058, a Bluetooth 2002 (BLE) 1059, a Bluetooth 2004 (BLE) 1060, a Bluetooth 2006 (BLE) 1070, a Bluetooth 2008 (BLE) 1072, a Bluetooth 2009 (BLE) 1080, a Bluetooth 2009 (BLE) 1090, a Bluetooth 2001 (BLE) 1092, a Bluetooth 2002 (BLE) 1094, a Bluetooth 2003 (BLE) 1096, a Bluetooth 2004 (BLE) 1098, a Bluetooth 2005 (BLE) 1099, a Bluetooth 2006 (BLE) 1091, a Bluetooth 2007 (BLE) 1092, a Bluetooth 2008 (BLE) 1091, a Bluetooth 2009 (BLE) 1092, a Bluetooth 2009 (BLE) 1094, a Bluetooth 2009 (BLE) 1095, a Bluetooth 2 The system may include a GPS (Global Positioning System) 1056, a Global Positioning System (GPS) 1055, a camera, such as a USB 3.0 camera ("USB 3.0 Camera") 1054, and / or a Low Power Double Data Rate ("LPDDR") memory unit ("LPDDR3") 1015, implemented, for example, to the LPDDR3 standard. Each of these components may be implemented in any suitable manner.

[0118] In at least one embodiment, other components may be communicatively coupled to the processor 1010 via the components described above. In at least one embodiment, an accelerometer 1041, an ambient light sensor ("ALS") 1042, a compass 1043, and a gyroscope 1044 may be communicatively coupled to the sensor hub 1040. In at least one embodiment, a thermal sensor 1039, a fan 1037, a keyboard 1036, and a touch pad 1030 may be communicatively coupled to the EC 1035. In at least one embodiment, a speaker 1063, a headphone 1064, and a microphone ("mic") 1065 may be communicatively coupled to an audio unit (audio codec and class d amplifier) ​​1062, which may be communicatively coupled to the DSP 1060. In at least one embodiment, the audio unit 1064 may include, for example, without limitation, an audio coder / decoder ("codec") and a class D amplifier. In at least one embodiment, a SIM card ("SIM") 1057 may be communicatively coupled to the WWAN unit 1056. In at least one embodiment, components such as the WLAN unit 1050 and the Bluetooth unit 1052, as well as the WWAN unit 1056, may be implemented in a Next Generation Form Factor ("NGFF").

[0119] Inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. More details regarding inference and / or training logic 715 are provided below in conjunction with Figures 7A and / or 7B. In at least one embodiment, inference and / or training logic 715 may be used in the system of Figure 10 for inference or prediction operations based at least in part on weight parameters calculated using neural network training operations, neural network functionality and / or architecture, or neural network use cases described herein.

[0120] These components can be used to generate synthetic data that mimics failure cases in the network training process, which may help improve the performance of the network while limiting the amount of synthetic data to avoid overfitting.

[0121] 11 is a block diagram of a processing system according to at least one embodiment. In at least one embodiment, system 1100 includes one or more processors 1102 and one or more graphics processors 1108 and may be a single processor desktop system, a multiprocessor workstation system, or a server system having multiple processors 1102 or processor cores 1107. In at least one embodiment, system 1100 is a processing platform integrated into a system-on-chip (SoC) integrated circuit for use in a mobile, portable, edge, or embedded device.

[0122] In at least one embodiment, the system 1100 may include or be incorporated into a server-based gaming platform, a game console including a game and media console, a mobile gaming console, a handheld game console, or an online game console. In at least one embodiment, the system 1100 is a mobile phone, a smart phone, a tablet computing device, or a mobile Internet device. In at least one embodiment, the processing system 1100 may also include, be coupled to, or be integrated into a wearable device, such as a smart watch wearable device, a smart eyewear device, an augmented reality device, or a virtual reality device. In at least one embodiment, the processing system 1100 is a television or set-top box device having one or more processors 1102 and a graphical interface generated by one or more graphics processors 1108.

[0123] In at least one embodiment, the one or more processors 1102 each include one or more processor cores 1107 for processing instructions that, when executed, perform operations for system and user software. In at least one embodiment, each of the one or more processor cores 1107 is configured to process a particular instruction set 1109. In at least one embodiment, the instruction set 1109 may facilitate complex instruction set computing (CISC), reduced instruction set computing (RISC), or computing via very long instruction words (VLIW). In at least one embodiment, the processor cores 1107 each may process a different instruction set 1109, which may include instructions that facilitate emulation of other instruction sets. In at least one embodiment, the processor cores 1107 may also include other processing devices, such as digital signal processors (DSPs).

[0124] In at least one embodiment, the processor 1102 includes a cache memory 1104. In at least one embodiment, the processor 1102 may have a single internal cache or multiple levels of internal cache. In at least one embodiment, the cache memory is shared among various components of the processor 1102. In at least one embodiment, the processor 1102 also uses an external cache (e.g., a level 3 (L3) cache or a last level cache (LLC)) (not shown), which may be shared among the processor cores 1107 using known cache coherence techniques. In at least one embodiment, the processor 1102 further includes a register file 1106, which may include different types of registers (e.g., integer registers, floating point registers, status registers, and an instruction pointer register) for storing different types of data. In at least one embodiment, the register file 1106 may include general purpose registers or other registers.

[0125] In at least one embodiment, the one or more processors 1102 are coupled to one or more interface buses 1110 to transmit communication signals, such as address, data, or control signals, between the processors 1102 and other components in the system 1100. In at least one embodiment, the interface bus 1110 may be a processor bus, such as, in one embodiment, a version of a Direct Media Interface (DMI) bus. In at least one embodiment, the interface 1110 is not limited to a DMI bus, but may include one or more peripheral component interconnect buses (e.g., PCI, PCI Express), a memory bus, or other types of interface buses. In at least one embodiment, the processor 1102 includes an integrated memory controller 1116, and a platform controller hub 1130. In at least one embodiment, memory controller 1116 facilitates communication between memory devices and other components of system 1100, while platform controller hub (PCH) 1130 provides connection to I / O devices via a local I / O bus.

[0126] In at least one embodiment, the memory device 1120 may be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, a flash memory device, a phase change memory device, or any other memory device having suitable performance to serve as process memory. In at least one embodiment, the memory device 1120 may act as system memory for the system 1100, storing data 1122 and instructions 1121 for use by the one or more processors 1102 in executing applications or processes. In at least one embodiment, the memory controller 1116 is also coupled to an optional external graphics processor 1112, which may communicate with one or more graphics processors 1108 within the processor 1102 to perform graphics and media operations. In at least one embodiment, a display device 1111 may be connected to the processor 1102. In at least one embodiment, the display device 1111 may include one or more of an internal display device, such as a mobile electronic device or laptop device, or an external display device attached via a display interface (e.g., a display port, etc.). In at least one embodiment, the display device 1111 may include a head mounted display (HMD), such as a stereoscopic display device for use in virtual reality (VR) or augmented reality (AR) applications.

[0127] In at least one embodiment, the platform controller hub 1130 allows peripherals to connect to the memory device 1120 and the processor 1102 via a high-speed I / O bus. In at least one embodiment, the I / O peripherals include, but are not limited to, an audio controller 1146, a network controller 1134, a firmware interface 1128, a wireless transceiver 1126, a touch sensor 1125, and a data storage device 1124 (e.g., hard disk drive, flash memory, etc.). In at least one embodiment, the data storage device 1124 may connect via a storage interface (e.g., SATA) or via a peripheral bus such as a peripheral component interconnect bus (e.g., PCI, PCI Express). In at least one embodiment, the touch sensor 1125 may include a touch screen sensor, a pressure sensor, or a fingerprint sensor. In at least one embodiment, wireless transceiver 1126 may be a Wi-Fi transceiver, a Bluetooth transceiver, or a mobile network transceiver such as a 3G, 4G, or Long Term Evolution (LTE) transceiver. In at least one embodiment, firmware interface 1128 enables communication with system firmware and may be, for example, a unified extensible firmware interface (UEFI). In at least one embodiment, network controller 1134 may enable network connectivity to a wired network. In at least one embodiment, a high performance network controller (not shown) couples to interface bus 1110. In at least one embodiment, audio controller 1146 is a multi-channel high definition audio controller. In at least one embodiment, system 1100 includes an optional legacy I / O controller 1140 for coupling legacy (e.g., Personal System 2 (PS / 2)) devices to the system.In at least one embodiment, platform controller hub 1130 may also connect to one or more universal serial bus (USB) controllers 1142 for connecting input devices, such as a keyboard and mouse 1143 combination, a camera 1144, or other USB input devices.

[0128] In at least one embodiment, instances of memory controller 1116 and platform controller hub 1130 may be integrated into a separate external graphics processor, such as external graphics processor 1111. In at least one embodiment, platform controller hub 1130 and / or memory controller 1116 may be external to one or more processors 1102. For example, in at least one embodiment, system 1100 may include an external memory controller 1116 and platform controller hub 1130, which may be configured as a memory controller hub and a peripheral controller hub in a system chipset that communicates with processor 1102.

[0129] Inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. More details regarding the inference and / or training logic 715 are provided below in conjunction with FIG. 7A and / or FIG. 7B. In at least one embodiment, some or all of the inference and / or training logic 715 may be incorporated into the graphics processor 1108. For example, in at least one embodiment, the training and / or inference techniques described herein may use one or more of the ALUs embodied in the graphics processor. Furthermore, in at least one embodiment, the inference and / or training operations described herein may be performed using logic other than that shown in FIG. 7A or FIG. 7B. In at least one embodiment, the weight parameters may be stored in on-chip or off-chip memory and / or registers (as shown or not shown) that configure the ALUs of the graphics processor to execute one or more machine learning algorithms, neural network architectures, use cases, or training techniques described herein.

[0130] These components can be used to generate synthetic data that mimics failure cases in the network training process, which may help improve the performance of the network while limiting the amount of synthetic data to avoid overfitting.

[0131] 12 is a block diagram of a processor 1200 having one or more processor cores 1202A-1202N, an integrated memory controller 1213, and an integrated graphics processor 1208, according to at least one embodiment. In at least one embodiment, processor 1200 may include additional cores, including additional core 1202N, represented by a dashed box. In at least one embodiment, each of processor cores 1202A-1202N includes one or more internal cache units 1204A-1204N. In at least one embodiment, each processor core also has access to one or more shared cache units 1206.

[0132] In at least one embodiment, the internal cache units 1204A-1204N and the shared cache unit 1206 represent a cache memory hierarchy within the processor 1200. In at least one embodiment, the cache memory units 1204A-1204N may include at least one level of instruction and data cache within each processor core, as well as one or more levels of shared mid-level cache, such as level 2 (L2), level 3 (L3), level 4 (L4), or other levels of cache, where the highest level of cache before external memory is classified as an LLC. In at least one embodiment, cache coherence logic maintains coherency between the various cache units 1206 and 1204A-1204N.

[0133] In at least one embodiment, processor 1200 may also include a set of one or more bus controller units 1216 and a system agent core 1210. In at least one embodiment, one or more bus controller units 1216 manage a set of peripheral buses, such as one or more PCI or PCI Express buses. In at least one embodiment, system agent core 1210 provides management functions for various processor components. In at least one embodiment, system agent core 1210 includes one or more integrated memory controllers 1213 for managing access to various external memory devices (not shown).

[0134] In at least one embodiment, one or more of the processor cores 1202A-1202N include support for simultaneous multithreading. In at least one embodiment, the system agent core 1210 includes components for coordinating and operating the cores 1202A-1202N during multithreaded processing. In at least one embodiment, the system agent core 1210 may further include a power control unit (PCU), which includes logic and components for coordinating the power state of one or more of the processor cores 1202A-1202N and the graphics processor 1208.

[0135] In at least one embodiment, processor 1200 further includes a graphics processor 1208 for performing graphics processing operations. In at least one embodiment, graphics processor 1208 couples to shared cache unit 1206 and to a system agent core 1210 that includes one or more integrated memory controllers 1213. In at least one embodiment, system agent core 1210 also includes a display controller 1211 for directing output of the graphics processor to one or more coupled displays. In at least one embodiment, display controller 1211 may also be a separate module coupled to graphics processor 1208 via at least one interconnect or may be integrated within graphics processor 1208.

[0136] In at least one embodiment, a ring-based interconnect unit 1212 is used to couple the internal components of processor 1200. In at least one embodiment, alternative interconnect units such as point-to-point interconnects, switched interconnects, or other techniques may be used. In at least one embodiment, graphics processor 1208 couples to ring interconnect 1212 via I / O link 1213.

[0137] In at least one embodiment, I / O link 1213 represents at least one of a variety of I / O interconnects, including an on-package I / O interconnect that facilitates communication between various processor components and a high performance embedded memory module 1218, such as an eDRAM module. In at least one embodiment, each of processor cores 1202A-1202N and graphics processor 1208 use embedded memory module 1218 as a shared last level cache.

[0138] In at least one embodiment, the processor cores 1202A-1202N are homogenous cores that execute a common instruction set architecture. In at least one embodiment, the processor cores 1202A-1202N are heterogeneous in terms of instruction set architecture (ISA), where one or more of the processor cores 1202A-1202N execute a common instruction set, while one or more other of the processor cores 1202A-1202N execute a subset of the common instruction set, or a different instruction set. In at least one embodiment, the processor cores 1202A-1202N are heterogeneous in terms of microarchitecture, where one or more cores with relatively higher power consumption are combined with one or more cores with lower power consumption. In at least one embodiment, the processor 1200 may be implemented on one or more chips or as an SoC integrated circuit.

[0139] Inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. More details regarding the inference and / or training logic 715 are provided below in conjunction with FIG. 7A and / or FIG. 7B. In at least one embodiment, some or all of the inference and / or training logic 715 may be incorporated into the processor 1200. For example, in at least one embodiment, the training and / or inference techniques described herein may use one or more of the ALUs embodied in the graphics processor 1208, the graphics cores 1202A-1202N, or other components of FIG. 12. Furthermore, in at least one embodiment, the inference and / or training operations described herein may be performed using logic other than that shown in FIG. 7A or FIG. 7B. In at least one embodiment, the weight parameters may be stored in on-chip or off-chip memory and / or registers (as shown or not shown) that configure the ALUs of the graphics processor 1200 to perform one or more machine learning algorithms, neural network architectures, use cases, or training techniques described herein.

[0140] These components can be used to generate synthetic data that mimics failure cases in the network training process, which may help improve the performance of the network while limiting the amount of synthetic data to avoid overfitting.

[0141] FIG. 13 is an exemplary data flow diagram of a process 1300 for generating and deploying an image processing and inference pipeline, according to at least one embodiment. In at least one embodiment, the process 1300 may be deployed for use with imaging devices, processing devices, and / or other types of devices at one or more facilities 1302. The process 1300 may be performed in a training system 1304 and / or in a deployment system 1306. In at least one embodiment, the training system 1304 may be used to train, deploy, and implement machine learning models (e.g., neural networks, object detection algorithms, computer vision algorithms, etc.) for use in the deployment system 1306. In at least one embodiment, the deployment system 1306 may be configured to offload processing and computational resources between distributed computing environments to reduce infrastructure requirements at the facility 1302. In at least one embodiment, one or more applications in the pipeline may use or call services (e.g., inference, virtualization, computation, AI, etc.) of the deployment system 1306 during execution of the applications.

[0142] In at least one embodiment, some of the applications used in the advanced processing and inference pipeline may use machine learning models or other AI to perform one or more processing steps. In at least one embodiment, the machine learning models may be trained at the facility 1302 using data (e.g., imaging data) 1308 generated at the facility 1302 (and stored in one or more picture archiving and communication system (PACS) servers at the facility 1302), may be trained using imaging or sequencing data 1308 from one or more other facilities, or a combination thereof. In at least one embodiment, a training system 1304 may be used to provide applications, services, and / or other resources for generating practical, deployable machine learning models for a deployment system 1306.

[0143] In at least one embodiment, the model registry 1324 may be backed by an object storage that may support versioning and object metadata. In at least one embodiment, the object storage may be accessible, for example, from within a cloud platform, via a cloud storage (e.g., cloud 1426 of FIG. 14) compatibility application programming interface (API). In at least one embodiment, machine learning models in the model registry 1324 may be uploaded, listed, modified, or deleted by a system developer or partner interacting with the API. In at least one embodiment, the API may provide access to a method that allows a user with appropriate credentials to associate a model with an application, thereby allowing the model to be executed as part of running a containerized instance of the application.

[0144] In at least one embodiment, the training pipeline 1404 (FIG. 14) may include a scenario in which the facility 1302 is training its own machine learning model or has an existing machine learning model that needs to be optimized or updated. In at least one embodiment, imaging data 1308 generated by an imaging device, a sequencing device, and / or other type of device may be received. In at least one embodiment, once the imaging data 1308 is received, AI-assisted annotation 1310 may be used to assist in generating annotations corresponding to the imaging data 1308 that will be used as ground truth data for the machine learning model. In at least one embodiment, the AI-assisted annotation 1310 may include one or more machine learning models (e.g., a convolutional neural network (CNN)), which may be trained to generate annotations corresponding to a particular type of imaging data 1308 (e.g., from a particular device). In at least one embodiment, the AI-assisted annotation 1310 may then be used directly to generate ground truth data or may be adjusted or fine-tuned using an annotation tool. In at least one embodiment, the AI-assisted annotations 1310, the labeled clinic data 1312, or a combination thereof may be used as ground truth data to train a machine learning model. In at least one embodiment, the trained machine learning model may be referred to as an output model 1316 and may be used by the deployment system 1306 described herein.

[0145] In at least one embodiment, the training pipeline 1404 (FIG. 14) may include a scenario in which the facility 1302 needs a machine learning model to use in performing one or more processing tasks for one or more applications in the installation system 1306, but the facility 1302 may not currently have such a machine learning model (or may not have a model that is optimized, efficient, or effective for such purposes). In at least one embodiment, an existing machine learning model may be selected from the model registry 1324. In at least one embodiment, the model registry 1324 may include machine learning models that have been trained to perform a variety of different inference tasks on imaging data. In at least one embodiment, the machine learning models in the model registry 1324 may have been trained on imaging data from a facility different from the facility 1302 (e.g., a facility in a remote location). In at least one embodiment, the machine learning models may have been trained on imaging data from one location, two locations, or any number of locations. In at least one embodiment, when trained on imaging data from a particular location, the training may be performed at the location, or at least in a manner that protects the confidentiality of the imaging data or limits the imaging data from being transferred off-premise. In at least one embodiment, once a model is trained or partially trained at one location, the machine learning model may be added to a model registry 1324. In at least one embodiment, the machine learning model may then be retrained or updated at any number of other facilities, and the retrained or updated model may be made available in the model registry 1324. In at least one embodiment, the machine learning model may then be selected from the model registry 1324, and may be referred to as an output model 1316, and may be used in the installation system 1306 to perform one or more processing tasks for one or more applications of the installation system.

[0146] In at least one embodiment, a scenario of the training pipeline 1404 (FIG. 14) may include a situation in which the facility 1302 needs a machine learning model to use in performing one or more processing tasks for one or more applications in the deployment system 1306, but the facility 1302 may not currently have such a machine learning model (or may not have a model optimized, efficient, or effective for such purposes). In at least one embodiment, the machine learning model selected from the model registry 1324 may not be fine-tuned or optimized for the imaging data 1308 generated at the facility 1302 due to differences in the population, robustness of the training data used to train the machine learning model, variety of anomalies in the training data, and / or other issues with the training data. In at least one embodiment, AI-assisted annotation 1310 may be used to assist in generating annotations corresponding to the imaging data 1308 to be used as ground truth data for retraining or updating the machine learning model. In at least one embodiment, the labeled data 1312 may be used as ground truth data for training the machine learning model. In at least one embodiment, retraining or updating the machine learning model may be referred to as model training 1314. In at least one embodiment, the model training 1314, e.g., the AI-assisted annotations 1310, the labeled clinic data 1312, or a combination thereof, may be used as ground truth data to retrain or update the machine learning model. In at least one embodiment, the trained machine learning model may be referred to as output model 1316 and may be used by the deployment system 1306 described herein.

[0147] In at least one embodiment, the deployment system 1306 may include software 1318, services 1320, hardware 1322, and / or other components, features, and functions. In at least one embodiment, the deployment system 1306 may include a software “stack” whereby the software 1318 may be built on top of the services 1320 and may use the services 1320 to perform some or all of the processing tasks, and the services 1320 and the software 1318 may be built on top of the hardware 1322 and may use the hardware 1322 to perform the processing, storage, and / or other computational tasks of the deployment system 1306. In at least one embodiment, the software 1318 may include any number of different containers, where each container may perform an instantiation of an application. In at least one embodiment, each application may perform one or more processing tasks (e.g., inference, object detection, feature detection, segmentation, image enhancement, calibration, etc.) of an advanced processing and inference pipeline. In at least one embodiment, an advanced processing and inference pipeline may be defined based on a selection of different containers desired or required to process the imaging data 1308, in addition to the containers that receive and compose the imaging data used by each container and / or used by facility 1302 after processing through the pipeline (e.g., to reconvert the output into a usable type of data). In at least one embodiment, the combination of containers in software 1318 (e.g., comprising a pipeline) may be referred to as a virtual device (described in more detail herein), which may utilize services 1320 and hardware 1322 to perform some or all of the processing tasks of the applications instantiated in the containers.

[0148] In at least one embodiment, the data processing pipeline may receive input data (e.g., imaging data 1308) in a particular format in response to an inference request (e.g., a request from a user of the deployment system 1306). In at least one embodiment, the input data may represent one or more images, videos, and / or other data representations generated by one or more imaging devices. In at least one embodiment, the data may undergo pre-processing as part of the data processing pipeline to prepare the data for processing by one or more applications. In at least one embodiment, post-processing may be performed on the output of one or more inference or other processing tasks of the pipeline to prepare output data for a subsequent application and / or for transmission and / or use by a user (e.g., in response to an inference request). In at least one embodiment, the inference tasks may be performed by one or more machine learning models, such as trained or deployed neural networks, which may include the output model 1316 of the training system 1304.

[0149] In at least one embodiment, tasks in a data processing pipeline may be encapsulated in containers, where each container represents a separate, fully functional instantiation of an application and a virtualized computing environment that can reference a machine learning model. In at least one embodiment, containers or applications may be published to a private (e.g., restricted access) area of ​​a container registry (described in more detail herein), and trained or deployed models may be stored in a model registry 1324 and associated with one or more applications. In at least one embodiment, images of the applications (e.g., images of containers) may be available in the container registry, and once selected from the container registry by a user for deployment in the pipeline, the images may be used to generate a container for instantiating the application for use on the user's system.

[0150] In at least one embodiment, a developer (e.g., a software developer, clinician, doctor, etc.) may develop, publish, and store an application (e.g., as a container) to perform image processing and / or inference on the provided data. In at least one embodiment, the development, publishing, and / or storage may be performed using a software development kit (SDK) associated with the system (e.g., to ensure that the developed application and / or container is compliant with or compatible with the system). In at least one embodiment, the developed application may be tested locally (e.g., at the first facility, on data from the first facility) using the SDK that can support at least a portion of the services 1320 as a system (e.g., system 1400 of FIG. 14). In at least one embodiment, because a DICOM object can contain anywhere from one to hundreds of images or other types of data, and there is variation in the data, the developer may be responsible for managing the extraction and preparation of the incoming data (e.g., setting configurations for the application, building pre-processing into the application, etc.). In at least one embodiment, once verified (e.g., for accuracy) by system 1400, the application may be made available in a container registry for selection and / or implementation by a user, and one or more processing tasks may be performed on the data at the user's facility (e.g., a second facility).

[0151] In at least one embodiment, the developer may then share the application or container over a network for access and use by users of the system (e.g., system 1400 of FIG. 14). In at least one embodiment, the completed and validated application or container may be stored in a container registry, and the associated machine learning model may be stored in a model registry 1324. In at least one embodiment, a requesting entity issuing an inference or image processing request may browse the container registry and / or model registry 1324 to find applications, containers, datasets, machine learning models, etc., select a desired combination of elements for inclusion in a data processing pipeline, and submit an image processing request. In at least one embodiment, the request may include input data (and in some instances, associated patient data) required to execute the request, and / or may include a selection of an application and / or machine learning model to be executed in processing the request. In at least one embodiment, the request may then be passed to one or more components of the deployment system 1306 (e.g., cloud) to execute the processing of the data processing pipeline. In at least one embodiment, processing by the deployment system 1306 may include referencing selected elements (e.g., applications, containers, models, etc.) from the container registry and / or the model registry 1324. In at least one embodiment, once results are produced by the pipeline, the results may be returned to a user for viewing (e.g., viewed locally in a viewing application suite running on a workstation or terminal on premises).

[0152] In at least one embodiment, services 1320 may be utilized to assist in the processing or execution of applications or containers in the pipeline. In at least one embodiment, services 1320 may include computational services, artificial intelligence (AI) services, visualization services, and / or other types of services. In at least one embodiment, services 1320 may provide common functionality to one or more applications of software 1318, whereby functionality may be abstracted into services that may be called or utilized by the applications. In at least one embodiment, functionality provided by services 1320 may be dynamically and more efficiently performed while at the same time scaling well by allowing applications to process data in parallel (e.g., using parallel computing platform 1430 (FIG. 14)). In at least one embodiment, services 1320 may be shared among various applications, rather than requiring each application that shares the same functionality provided by services 1320 to have a respective instance of services 1320. In at least one embodiment, services may include, by way of non-limiting example, inference servers or engines that may be used to perform detection or segmentation tasks. At least one embodiment may include a model training service that may provide training and / or retraining of machine learning models. At least one embodiment may further include a data augmentation service that may provide GPU accelerated data (e.g., DICOM, RIS, CIS, REST-compliant, RPC, raw, etc.) extraction, resizing, scaling, and / or other enhancements. At least one embodiment may include a visualization service that may add image rendering effects such as ray tracing, rasterization, denoising, sharpening, etc. to add realism to two-dimensional (2D) and / or three-dimensional (3D) models. At least one embodiment may include a virtual device service that provides beamforming, segmentation, inference, imaging, and / or support for other applications in the virtual device pipeline.

[0153] In at least one embodiment, if the services 1320 include an AI service (e.g., an inference service), the one or more machine learning models may be executed by calling (as an API call) to an inference service (e.g., an inference server) to execute the machine learning models, or processing thereof, as part of the application execution. In at least one embodiment, if another application includes one or more machine learning models for a segmentation task, the application may call the inference service to execute the machine learning models to perform one or more of the processing operations associated with the segmentation task. In at least one embodiment, software 1318 implementing advanced processing and inference pipelines, including a segmentation application and an anomaly detection application, may be streamlined since each application may call the same inference service to perform one or more inference tasks.

[0154] In at least one embodiment, the hardware 1322 may include a GPU, a CPU, a DPU, a graphics card, an AI / deep learning system (e.g., an AI supercomputer such as NVIDIA's DGX), a cloud platform, or a combination thereof. In at least one embodiment, different types of hardware 1322 may be used to provide efficient and dedicated support for the software 1318 and services 1320 of the deployment system 1306. In at least one embodiment, the use of GPU processing to perform processing locally (e.g., at the facility 1302), within the AI / deep learning system, in the cloud system, and / or in other processing components of the deployment system 1306 to improve the efficiency, accuracy, and effectiveness of image processing and image generation. In at least one embodiment, the software 1318 and / or services 1320 may be optimized for GPU processing for deep learning, machine learning, and / or high performance computing, as non-limiting examples. In at least one embodiment, at least a portion of the computing environment of the deployment system 1306 and / or the training system 1304 may be executed in one or more supercomputers or high performance computing systems in a data center using GPU-optimized software (e.g., a hardware and software combination of NVIDIA's DGX system). In at least one embodiment, the hardware 1322 may include any number of GPUs, which may be called upon to perform parallel processing of data as described herein. In at least one embodiment, the cloud platform may further include GPU processing for GPU-optimized execution of deep learning tasks, machine learning tasks, or other computing tasks. In at least one embodiment, the cloud platform may further include DPU processing for sending data received over a network and / or via a network controller or other network interface directly to one or more GPUs (e.g., memory of the GPUs).In at least one embodiment, a cloud platform (e.g., NVIDIA's NGC) may run AI / deep learning supercomputers (e.g., provided by NVIDIA's DGX systems) and / or GPU-optimized software as a hardware abstraction and scaling platform. In at least one embodiment, the cloud platform may integrate an application container clustering system or orchestration system (e.g., KUBERNETES) for multiple GPUs to enable seamless scaling and load balancing.

[0155] 14 is a system diagram illustrating an example system 1400 for generating and deploying an imaging deployment pipeline, according to at least one embodiment. In at least one embodiment, system 1400 may be used to implement process 1300 of FIG. 13 and / or other processes including advanced processing and inference pipelines. In at least one embodiment, system 1400 may include training system 1304 and deployment system 1306. In at least one embodiment, training system 1304 and deployment system 1306 may be implemented using software 1318, services 1320, and / or hardware 1322, as described herein.

[0156] In at least one embodiment, the system 1400 (e.g., the training system 1304 and / or the deployment system 1306) may be implemented in a cloud computing environment (e.g., the cloud 1426). In at least one embodiment, the system 1400 may be implemented locally with respect to a healthcare service facility or as a combination of cloud and local computing resources. In at least one embodiment, access to the APIs of the cloud 1426 may be limited to authorized users via enacted security measures or protocols. In at least one embodiment, the security protocol may include a web token, which may be signed by an authentication (e.g., AuthN, AuthZ, Gluecon, etc.) service and may have appropriate permissions. In at least one embodiment, the APIs of a virtual device (as described herein) or other instantiation of the system 1400 may be limited to a set of public IPs that have been verified or authorized for interaction.

[0157] In at least one embodiment, the various components of system 1400 may communicate with one another using any of a variety of different types of networks, including, but not limited to, a local area network (LAN) and / or a wide area network (WAN), via wired and / or wireless communication protocols. In at least one embodiment, communications between the facility and the components of system 1400 (e.g., for sending inference requests, receiving results of inference requests, etc.) may be communicated via one or more data buses, wireless data protocols (Wi-Fi), wired data protocols (e.g., Ethernet), etc.

[0158] In at least one embodiment, the training system 1304 may execute a training pipeline 1404 similar to that described herein with respect to FIG. 13. In at least one embodiment, if one or more machine learning models are to be used by the deployment system 1306 in the deployment pipeline 1410, the training pipeline 1404 may be used to train or retrain one or more (e.g., pre-trained) models and / or to implement one or more of the pre-trained models 1406 (e.g., without the need for retraining or updating). In at least one embodiment, the training pipeline 1404 may result in an output model 1316. In at least one embodiment, the training pipeline 1404 may include any number of processing steps, such as, but not limited to, transforming or adapting the imaging data (or other input data). In at least one embodiment, different training pipelines 1404 may be used for different machine learning models used by the deployment system 1306. In at least one embodiment, a training pipeline 1404 similar to the first example described with respect to FIG. 13 may be used for the first machine learning model, a training pipeline 1404 similar to the second example described with respect to FIG. 13 may be used for the second machine learning model, and a training pipeline 1404 similar to the third example described with respect to FIG. 13 may be used for the third machine learning model. In at least one embodiment, any combination of tasks in the training system 1304 may be used depending on what is required for each respective machine learning model. In at least one embodiment, one or more of the machine learning models may already be trained and ready for deployment, such that the machine learning models do not need to undergo any processing by the training system 1304 and may be implemented by the deployment system 1306.

[0159] In at least one embodiment, the output model 1316 and / or the pre-trained model 1406 may include any type of machine learning model depending on the implementation or embodiment. In at least one embodiment, without limitation, the machine learning models used by the system 1400 may include machine learning models using linear regression, logistic regression, decision trees, support vector machines (SVMs), naive Bayes, k-nearest neighbor (Knn), k-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoders, convolutional, recurrent, perceptrons, long / short term memory (LSTM), Hopfield, Boltzmann, deep belief, deconvolution, generative adversarial, liquid state machines, etc.), and / or other types of machine learning models.

[0160] In at least one embodiment, the training pipeline 1404 may include AI-assisted annotations, as described in more detail herein with respect to at least FIG. 16B. In at least one embodiment, the labeled data 1312 (e.g., traditional annotations) may be generated by any number of techniques. In at least one embodiment, the labels or other annotations may be generated within a drawing program (e.g., an annotation program), a computer-aided design (CAD) program, a labeling program, another type of program suitable for generating annotations or labels for ground truth, and / or may be hand-drawn in some instances. In at least one embodiment, the ground truth data may be synthetically generated (e.g., generated from a computer model or rendering), realistically generated (e.g., designed and generated from real-world data), machine-automated (e.g., using feature analysis and learning to extract features from data and then generate labels), human-annotated (e.g., a labeler or annotation expert may define the location of the labels), and / or a combination thereof. In at least one embodiment, for each instance of imaging data 1308 (or other type of data used by the machine learning model), there may be corresponding ground truth data generated by the training system 1304. In at least one embodiment, AI-assisted annotation may be performed as part of the deployment pipeline 1410 in addition to or instead of AI-assisted annotation included in the training pipeline 1404. In at least one embodiment, the system 1400 may include a multi-tiered platform that may include a software layer (e.g., software 1318) of a diagnostic application (or other type of application) that may perform one or more medical imaging and diagnostic functions. In at least one embodiment, the system 1400 may be communicatively coupled (e.g., via an encrypted link) to one or more facility's PACS server networks.In at least one embodiment, system 1400 may be configured to access and reference data from a PACS server to perform operations such as training machine learning models, deploying machine learning models, image processing, inference, and / or other operations.

[0161] In at least one embodiment, the software layer may be implemented as a secure, encrypted, and / or authenticated API through which an application or container may be invoked (e.g., called) from an external environment (e.g., facility 1302). In at least one embodiment, the applications may then call or execute one or more services 1320 to perform computational, AI, or visualization tasks associated with the respective application, and the software 1318 and / or services 1320 may utilize hardware 1322 to perform the processing tasks in an effective and efficient manner.

[0162] In at least one embodiment, the introduction system 1306 may execute an introduction pipeline 1410. In at least one embodiment, the introduction pipeline 1410 may include any number of applications, including the AI-assisted annotations described above, that may be applied sequentially, non-sequentially, or otherwise to imaging data (and / or other types of data) generated by an imaging device, a sequencing device, a genomics device, etc. In at least one embodiment, as described herein, the introduction pipeline 1410 for an individual device may be referred to as a virtual device for the device (e.g., a virtual ultrasound device, a virtual CT scan device, a virtual sequencing device, etc.). In at least one embodiment, there may be more than one introduction pipeline 1410 per device depending on the information required for the data generated by the device. In at least one embodiment, if anomaly detection is required for the MRI machine, a first introduction pipeline 1410 may be present, and if image enhancement is required for the output of the MRI machine, a second introduction pipeline 1410 may be present.

[0163] In at least one embodiment, the image generation application may include processing tasks that include the use of machine learning models. In at least one embodiment, a user may wish to use their own machine learning models or select a machine learning model from the model registry 1324. In at least one embodiment, a user may implement their own machine learning models or select a machine learning model to include in the application to perform the processing tasks. In at least one embodiment, the applications may be selectable and customizable, and by defining the structure of the application, the deployment and implementation of the application for a particular user may be presented as a more seamless user experience. In at least one embodiment, by utilizing other features of the system 1400, such as the services 1320 and the hardware 1322, the deployment pipeline 1410 may become even more user friendly, may provide easier integration, and may produce more accurate, efficient, and timely results.

[0164] In at least one embodiment, the deployment system 1306 may include a user interface 1414 (e.g., a graphical user interface, a web interface, etc.), which may be used to select applications for inclusion in the deployment pipeline 1410, configure applications, modify or change applications or their parameters or structure, use and interact with the deployment pipeline 1410 during setup and / or deployment, and / or otherwise interact with the deployment system 1306. In at least one embodiment, while not shown with respect to the training system 1304, the user interface 1414 (or a different user interface) may be used to select models for use in the deployment system 1306, to select models to train or retrain in the training system 1304, and / or to otherwise interact with the training system 1304.

[0165] In at least one embodiment, a pipeline manager 1412 may be used in addition to an application orchestration system 1428 to manage interactions between applications or containers in the deployment pipeline 1410 and services 1320 and / or hardware 1322. In at least one embodiment, the pipeline manager 1412 may be configured to facilitate application-to-application interactions, application-to-service interactions, and / or application or service interactions to hardware 1322. While illustrated in at least one embodiment as being included in software 1318, this is not intended to be limiting, and in some instances (e.g., such as those shown in FIG. 12 ), the pipeline manager 1412 may be included in services 1320. In at least one embodiment, the application orchestration system 1428 (e.g., Kubernetes, DOCKER, etc.) may include a container orchestration system that may group applications into containers as logical units for orchestration, management, scaling, and deployment. In at least one embodiment, by associating applications from the deployment pipeline 1410 (e.g., a reconstruction application, a segmentation application, etc.) with individual containers, each application can run in a self-contained environment (e.g., at the kernel level) to improve speed and efficiency.

[0166] In at least one embodiment, each application and / or container (or an image thereof) may be developed, modified, and deployed individually (e.g., a first user or developer may develop, modify, and deploy a first application, and a second user or developer may develop, modify, and deploy a second application separately from the first user or developer), allowing focus and attention to be paid to the tasks of one application and / or container without being distracted by the tasks of another application or container. In at least one embodiment, communication and coordination between different containers or applications may be facilitated by the pipeline manager 1412 and the application orchestration system 1428. In at least one embodiment, the application orchestration system 1428 and / or the pipeline manager 1412 can facilitate communication between, and sharing of resources between, each of the applications or containers, so long as the expected inputs and / or outputs of each container or application are known by the system (e.g., based on the structure of the application or container). In at least one embodiment, one or more of the applications or containers in the deployment pipeline 1410 may share the same services and resources, so the application orchestration system 1428 may orchestrate, load balance, and determine sharing of services or resources among the various applications or containers. In at least one embodiment, a scheduler may be used to track the resource requirements of the applications or containers, the current or planned usage of those resources, and the availability of the resources. In at least one embodiment, the scheduler may thus allocate resources to different applications and distribute resources among the applications taking into account the requirements and availability of the system.In some instances, the scheduler (and / or other components of the application orchestration system 1428) may determine resource availability and distribution based on constraints imposed on the system (e.g., user constraints), such as quality of service (QoS), the urgency of needing data output (e.g., to determine whether to perform real-time or deferred processing), etc.

[0167] In at least one embodiment, services 1320 utilized and shared by applications or containers of deployment system 1306 may include compute services 1416, AI services 1418, visualization services 1420, and / or other types of services. In at least one embodiment, an application may call (e.g., execute) one or more of services 1320 to perform processing operations for the application. In at least one embodiment, compute services 1416 may be utilized by an application to perform supercomputing or other high performance computing (HPC) tasks. In at least one embodiment, parallel processing may be performed utilizing compute services 1416 (e.g., using parallel computing platform 1430) to substantially simultaneously process data via one or more of the applications and / or to substantially simultaneously process one or more tasks of an application. In at least one embodiment, parallel computing platform 1430 (e.g., NVIDIA's CUDA) may enable general purpose computing on a GPU (GPGPU) (e.g., GPU 1422). In at least one embodiment, the software layer of the parallel computing platform 1430 may provide access to virtual instruction sets and parallel computational elements of a GPU to execute computational kernels. In at least one embodiment, the parallel computing platform 1430 may include memory, and in some embodiments, memory may be shared among multiple containers and / or among different processing tasks within a container. In at least one embodiment, inter-process communication (IPC) calls may be generated in order for multiple containers and / or multiple processes within a container to use the same data from a shared segment of memory of the parallel computing platform 1430 (e.g., when multiple different stages of an application or multiple applications process the same information).In at least one embodiment, rather than making copies of the data and moving the data to a different location in memory (e.g., read / write operations), the same data in the same location in memory may be used for any number of processing tasks (e.g., at the same time, at different times, etc.). In at least one embodiment, as data is used and new data is generated as a result of processing, this information of the new location of the data may be stored in and shared between various applications. In at least one embodiment, the location of the data and the location of updated or modified data may be part of the definition of how the payload is understood within the container.

[0168] In at least one embodiment, the AI ​​services 1418 may be utilized to perform inference services to execute machine learning models associated with an application (e.g., tasked with performing one or more processing tasks of the application). In at least one embodiment, the AI ​​services 1418 may utilize the AI ​​systems 1424 to execute machine learning models (e.g., neural networks such as CNNs) for segmentation, reconstruction, object detection, feature detection, classification, and / or other inference tasks. In at least one embodiment, an application in the deployment pipeline 1410 may perform inference on imaging data using one or more of the output models 1316 from the training system 1304 and / or other models of the application. In at least one embodiment, two or more instances of inference using the application orchestration system 1428 (e.g., a scheduler) may be available. In at least one embodiment, a first category may include a high priority / low latency path that can achieve higher service level agreements, such as for performing inference on urgent requests during an emergency or for a radiologist during a diagnosis. In at least one embodiment, the second category may include a standard priority path that may be used for non-urgent requests or when the analysis may be performed at a later time. In at least one embodiment, the application orchestration system 1428 may allocate resources (e.g., services 1320 and / or hardware 1322) based on the priority paths for different inference tasks of the AI ​​service 1418.

[0169] In at least one embodiment, a shared storage may be attached to the AI ​​service 1418 in the system 1400. In at least one embodiment, the shared storage may act as a cache (or other type of storage device) and may be used to process inference requests from applications. In at least one embodiment, when an inference request is sent, the request may be received by a set of API instances of the installation system 1306, and one or more instances may be selected (e.g., for best fit, for load balancing, etc.) to process the request. In at least one embodiment, to process the request, the request may be entered into a database, a machine learning model may be identified from the model registry 1324 if not already in the cache, and a validation step may ensure that the appropriate machine learning model is loaded into the cache (e.g., the shared storage), and / or a copy of the model may be saved to the cache. In at least one embodiment, if the application is not already running or if there are not enough instances of the application, a scheduler (e.g., the pipeline manager 1412) may be used to launch the application referenced in the request. In at least one embodiment, an inference server may be started to run the model if one has not already been started. Any number of inference servers may be started per model. In at least one embodiment, in a pull model where inference servers are clustered, models may be cached whenever load balancing is advantageous. In at least one embodiment, inference servers may be statically loaded onto corresponding distributed servers.

[0170] In at least one embodiment, inference may be performed using an inference server running within a container. In at least one embodiment, an instance of an inference server may be associated with a model (and optionally with multiple versions of the model). In at least one embodiment, when a request to perform inference on a model is received, if an instance of the inference server does not exist, a new instance may be loaded. In at least one embodiment, when the inference server is started, the model may be passed to the inference server, such that the same container may be used to serve different models, as long as the inference servers are running as different instances.

[0171] In at least one embodiment, while an application is running, an inference request may be received for a given application, a container (e.g., hosting an instance of an inference server) may be loaded (if not already loaded) and a start procedure may be called. In at least one embodiment, pre-processing logic of the container may load, decode, and / or perform any additional pre-processing on the input data (e.g., using a CPU and / or GPU and / or DPU). In at least one embodiment, once the data is prepared for inference, the container may perform inference on the data as needed. In at least one embodiment, this may include a single inference call for one image (e.g., a hand x-ray) or may require inference on hundreds of images (e.g., a chest CT). In at least one embodiment, the application may summarize the results before completion, which may include, without limitation, a single confidence score, pixel-level segmentation, voxel-level segmentation, generation of a visualization, or generation of text to summarize the findings. In at least one embodiment, different models or applications may be assigned different priorities. For example, some models may have real-time priority (TAT<1 minute) and other models may have low priority (e.g., TAT<12 minutes). In at least one embodiment, model execution time may be measured from the requesting facility or entity and may include execution to the inference service as well as partner network traversal time.

[0172] In at least one embodiment, the transition of requests between the service 1320 and the inference application may be hidden behind a software development kit (SDK) and robust transport may be provided through a queue. In at least one embodiment, requests are queued through an API for individual application / tenant ID combinations and the SDK pulls requests from the queue and gives them to the application. In at least one embodiment, a queue name may be provided in the environment where the SDK picks up the request. In at least one embodiment, asynchronous communication through the queue may be useful as it allows any instance of the application to pick up work when it is available. Results may be sent back through the queue to ensure that data is not lost. In at least one embodiment, queues may also provide the ability to segment work, as the highest priority work may go to a queue with the majority of instances of the application connected to the queue, while the lowest priority work may go to a queue with one instance connected to the queue that processes tasks in the order they are received. In at least one embodiment, applications may run on GPU accelerated instances spawned in the cloud 1426 and the inference service may run inference on the GPU.

[0173] In at least one embodiment, visualization services 1420 may be utilized to generate visualizations for viewing the applications and / or the output of the deployment pipeline 1410. In at least one embodiment, a GPU 1422 may be utilized by the visualization services 1420 to generate the visualizations. In at least one embodiment, rendering effects such as ray tracing may be implemented by the visualization services 1420 to generate higher quality visualizations. In at least one embodiment, visualizations may include, without limitation, 2D image rendering, 3D volume rendering, 3D volume reconstruction, 2D tomography slices, virtual reality displays, augmented reality displays, and the like. In at least one embodiment, virtualized environments may be used to generate virtual interactive displays or environments (e.g., virtual environments) for users of the system (e.g., doctors, nurses, radiologists, etc.) to interact with. In at least one embodiment, the visualization services 1420 may include internal visualizers, cinematics, and / or other rendering or image processing capabilities or functions (e.g., ray tracing, rasterization, internal optics, etc.).

[0174] In at least one embodiment, hardware 1322 may include GPU 1422, AI system 1424, cloud 1426, and / or any other hardware used to run training system 1304 and / or deployment system 1306. In at least one embodiment, GPU 1422 (e.g., NVIDIA TESLA and / or QUADRO GPUs) may include any number of GPUs that may be used to perform processing tasks for compute services 1416, AI services 1418, visualization services 1420, other services, and / or any features or functionality of software 1318. For example, with respect to AI services 1418, GPU 1422 may be used to perform pre-processing on imaging data (or other types of data used by machine learning models), post-processing on the output of machine learning models, and / or inference (e.g., machine learning models may be executed). In at least one embodiment, cloud 1426, AI system 1424, and / or other components of system 1400 may use GPU 1422. In at least one embodiment, cloud 1426 may include a GPU-optimized platform for deep learning tasks. In at least one embodiment, AI system 1424 may use GPUs, and cloud 1426, or at least a portion tasked with deep learning or inference, may be executed using one or more AI systems 1424. Thus, while hardware 1322 is shown as separate components, this is not intended to be limiting, and any component of hardware 1322 may be combined with or utilized by any other component of hardware 1322.

[0175] In at least one embodiment, the AI ​​system 1424 may include a specialized computing system (e.g., a supercomputer or HPC) configured for inference, deep learning, machine learning, and / or other artificial intelligence tasks. In at least one embodiment, the AI ​​system 1424 (e.g., NVIDIA's DGX) may include GPU-optimized software (e.g., a software stack) that may be executed using multiple GPUs 1422 in addition to DPUs, CPUs, RAM, storage, and / or other components, features, or functions. In at least one embodiment, one or more AI systems 1424 may be implemented in the cloud 1426 (e.g., in a data center) to perform some or all of the AI-based processing tasks of the system 1400.

[0176] In at least one embodiment, the cloud 1426 may include a GPU-accelerated infrastructure (e.g., NVIDIA's NGC), which may provide a GPU-optimized platform for executing the processing tasks of the system 1400. In at least one embodiment, the cloud 1426 may include an AI system 1424 (e.g., as a hardware abstraction and scaling platform) for executing one or more of the AI-based tasks of the system 1400. In at least one embodiment, the cloud 1426 may utilize multiple GPUs and integrate with an application orchestration system 1428 to enable seamless scaling and load balancing between applications and services 1320. In at least one embodiment, the cloud 1426 may be tasked with executing at least a portion of the services 1320 of the system 1400, including the compute services 1416, the AI ​​services 1418, and / or the visualization services 1420 described herein. In at least one embodiment, cloud 1426 may perform small or large batch inference (e.g., NVIDIA's Tensor RT implementation), may provide an accelerated parallel computing API and platform 1430 (e.g., NVIDIA's CUDA), may run an application orchestration system 1428 (e.g., KUBERNETES), may provide a graphics rendering API and platform (e.g., ray tracing, 2D graphics, 3D graphics, and / or other rendering techniques for generating high quality cinematics), and / or may provide other functionality for system 1400.

[0177] 15A illustrates a data flow diagram of a process 1500 for training, retraining, or updating a machine learning model, according to at least one embodiment. In at least one embodiment, the process 1500 may be performed using the system 1400 of FIG. 14 as a non-limiting example. In at least one embodiment, the process 1500 may utilize the services 1320 and / or hardware 1322 of the system 1400 described herein. In at least one embodiment, the refined model 1512 generated by the process 1500 may be executed by the deployment system 1306 for one or more containerized applications in the deployment pipeline 1410.

[0178] In at least one embodiment, model training 1314 may include retraining or updating the initial model 1504 (e.g., a pre-trained model) using new training data (e.g., new input data, such as the customer dataset 1506 and / or new ground truth data associated with the input data). In at least one embodiment, to retrain or update the initial model 1504, an output or loss layer of the initial model 1504 may be reset, removed, and / or replaced with an updated or new output or loss layer. In at least one embodiment, the initial model 1504 may have previously fine-tuned parameters (e.g., weights and / or biases) remaining from a previous training, such that training or retraining 1314 does not take as long or require as much processing as training a model from scratch. In at least one embodiment, during model training 1314, by having the output or loss layer of the initial model 1504 reset or replaced, parameters may be updated or retuned for the new data set based on a loss calculation associated with the accuracy of the output or loss layer in generating predictions for the new customer data set 1506 (e.g., image data 1308 in FIG. 13 ).

[0179] In at least one embodiment, the pre-trained model 1406 may be stored in a data store or registry (e.g., model registry 1324 of FIG. 13). In at least one embodiment, the pre-trained model 1406 may be trained, at least in part, at one or more facilities different from the facility performing the process 1500. In at least one embodiment, to protect the privacy and rights of patients, subjects, or customers of the different facilities, the pre-trained model 1406 may have been trained on-premise using customer or patient data generated on-premise. In at least one embodiment, the pre-trained model 1406 may be trained using the cloud 1426 and / or other hardware 1322, but the privacy-protected sensitive patient data may not be transferred to, used by, or accessible by any components of the cloud 1426 (or other off-premise hardware). In at least one embodiment, if the pre-trained model 1406 is trained using patient data from more than one facility, the pre-trained model 1406 may be trained individually for each facility and then trained on patient or customer data from another facility. In at least one embodiment, the pre-trained model 1406 may be trained on-premise and / or off-premise, such as in a data center or other cloud computing infrastructure, using customer or patient data from any number of facilities, such as if the customer or patient data is free of privacy concerns (e.g., via a waiver for experimental use) or if the customer or patient data is included in a public dataset.

[0180] In at least one embodiment, when selecting an application for use in the deployment pipeline 1410, the user may also select the machine learning model that will be used with the particular application. In at least one embodiment, the user may not have a model to use, and therefore the user may select a pre-trained model 1406 to use with the application. In at least one embodiment, the pre-trained model 1406 may be optimized to produce accurate results for the user's facility's customer dataset 1506 (e.g., based on patient diversity, demographics, type of medical imaging device used, etc.). In at least one embodiment, before the pre-trained model 1406 is deployed in the deployment pipeline 1410 for use with the application, the pre-trained model 1406 may be updated, re-trained, and / or fine-tuned for use at the respective facility.

[0181] In at least one embodiment, a user may select a pre-trained model 1406 to be updated, retrained, and / or fine-tuned, which may be referred to as an initial model 1504 for the training system 1304 in the process 1500. In at least one embodiment, model training 1314 (which may include, without limitation, transfer learning) may be performed on the initial model 1504 using a customer dataset 1506 (e.g., imaging data, genomics data, sequencing data, or other types of data generated by devices at the facility) to generate a refined model 1512. In at least one embodiment, ground truth data corresponding to the customer dataset 1506 may be generated by the training system 1304. In at least one embodiment, the ground truth data may be generated, at least in part, by clinicians, scientists, physicians, practitioners at the facility (e.g., as labeled clinic data 1312 in FIG. 13 ).

[0182] In at least one embodiment, AI-assisted annotation 1310 may be used in some instances to generate ground truth data. In at least one embodiment, AI-assisted annotation 1310 (e.g., implemented using an AI-assisted annotation SDK) may utilize machine learning models (e.g., neural networks) to generate suggested or predicted ground truth data for a customer dataset. In at least one embodiment, a user 1510 may use an annotation tool within a user interface (graphical user interface (GUI)) on a computing device 1508.

[0183] In at least one embodiment, a user 1510 may interact with a GUI via a computing device 1508 to edit or fine-tune the (automatic) annotation. In at least one embodiment, polygon editing features may be used to move vertices of polygons to more accurate or fine-tuned locations.

[0184] In at least one embodiment, once the customer dataset 1506 has associated ground truth data, the ground truth data (e.g., from AI-assisted annotation, manual labeling, etc.) may be used during model training 1314 to generate a refined model 1512. In at least one embodiment, the customer dataset 1506 may be applied to the initial model 1504 any number of times, and the ground truth data may be used to update parameters of the initial model 1504 until an acceptable level of accuracy is achieved for the refined model 1512. In at least one embodiment, once the refined model 1512 is generated, the refined model 1512 may be deployed into one or more deployment pipelines 1410 at the facility to perform one or more processing tasks on the medical imaging data.

[0185] In at least one embodiment, the refined model 1512 may be uploaded to the pre-trained models 1406 of the model registry 1324 to be selected by another facility. In at least one embodiment, this process may be completed at any number of facilities, such that the refined model 1512 may be further refined any number of times on new datasets to generate a more general model.

[0186] FIG. 15B is a diagram of an example client-server architecture 1532 for enhancing an annotation tool with a pre-trained annotation model, according to at least one embodiment. In at least one embodiment, an AI-assisted annotation tool 1536 may be instantiated based on the client-server architecture 1532. In at least one embodiment, the annotation tool 1536 of an imaging application may, for example, assist a radiologist in identifying organs and anomalies. In at least one embodiment, the imaging application may include, as a non-limiting example, a software tool that assists a user 1510 in identifying a few extreme points on a particular target organ in a raw image 1534 (e.g., of a 3D MRI or CT scan) and receives an automatically annotated result for all 2D slices of the particular organ. In at least one embodiment, the results may be stored in a data store as training data 1538 or may be used (for example, without limitation) as ground truth data for training. In at least one embodiment, when the computing device 1508 sends the extreme points for AI-assisted annotation 1310, for example, a deep learning model may receive this data as input and return an inference result of a segmented organ or anomaly. In at least one embodiment, a pre-instantiated annotation tool, such as the AI-assisted annotation tool 1536B of FIG. 15B, may be enhanced by making API calls (e.g., API calls 1544) to a server, such as an annotation assisted server 1540, which may include a set of pre-trained models 1542 stored in an annotation model registry. In at least one embodiment, the annotation model registry may store pre-trained models 1542 (e.g., machine learning models such as deep learning models) that have been pre-trained to perform AI-assisted annotation for a particular organ or anomaly. These models may be further updated by using the training pipeline 1404.In at least one embodiment, the pre-installed annotation tools may be improved over time as new labeled clinic data 1312 is added.

[0187] These components can be used to generate synthetic data that mimics failure cases in the network training process, which may help improve the performance of the network while limiting the amount of synthetic data to avoid overfitting.

[0188] Other variations are within the scope of the present disclosure. Thus, while the disclosed techniques are susceptible to various modifications and alternative constructions, certain illustrative examples thereof have been shown in the drawings and have been described in detail above. There is no intention, however, to limit the disclosure to the particular disclosed form or forms, but on the contrary, the intention is to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of the disclosure as defined by the appended claims.

[0189] The use of the terms "a" and "an" and "the" and similar referents in the context of describing the disclosed embodiments (particularly in the context of the claims that follow) should be construed to cover both the singular and the plural and should not be construed as defining terms, unless otherwise stated herein or clearly contradicted by context. The terms "comprising," "having," "including," and "containing" are to be construed as open-ended terms (meaning "including, but not limited to"), unless otherwise stated. "Connected," when referring to a physical connection without modification, is to be construed as being partially or completely contained within, attached to, or joined to one another, even if there is something intervening. The recitation of ranges of values ​​herein is merely intended to serve as a shorthand method of individually referring to each separate value falling within the range, unless otherwise stated herein and unless each separate value is incorporated herein as if it were individually recited herein. In at least one embodiment, use of the term "set" (e.g., "set of items") or "subset" should be construed as a non-empty collection of one or more elements, unless otherwise indicated or constrained by context. Furthermore, unless otherwise indicated or constrained by context, the term "subset" of a corresponding set does not necessarily refer to a strict subset of the corresponding set, and a subset and a corresponding set may be equivalent.

[0190] Conjunctions such as phrases of the form "at least one of A, B, and C" or "at least one of A, B, and C" are understood in the context in which they are generally used to indicate that an item, term, etc. is A, B, or C, or a non-empty subset of any of the sets A, B, and C, unless specifically stated otherwise or clearly contradicted by context. For example, in the illustrative example of a set having three elements, the conjunctive phrases "at least one of A, B, and C" and "at least one of A, B, and C" refer to any of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such conjunctives are not generally intended to indicate that a given embodiment requires the presence of at least one A, at least one B, and at least one C, respectively. Further, unless stated otherwise or clearly stated by context, the term "plurality" refers to a plurality (e.g., "a plurality of items" refers to multiple items). In at least one embodiment, the number of items in a plurality is at least two, but may be more if indicated explicitly or by context. Further, unless stated otherwise or clearly stated otherwise by context, the phrase "based on" means "based at least in part on" and not "based only on."

[0191] The operations of the processes described herein may be performed in any suitable order unless otherwise stated herein or clearly contradicted by context. In at least one embodiment, a process such as the processes described herein (or variations and / or combinations thereof) is implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) executed under the control of one or more computer systems configured with executable instructions and collectively executed on one or more processors, by hardware, or by a combination thereof. In at least one embodiment, the code is stored on a computer-readable storage medium, e.g., in the form of a computer program comprising a plurality of instructions executable by one or more processors. In at least one embodiment, the computer-readable storage medium is a non-transitory computer-readable storage medium that excludes transitory signals (e.g., propagating transitory electrical or electromagnetic transmissions), but includes non-transitory data storage circuitry (e.g., buffers, caches, and queues) within a transceiver of a transitory signal. In at least one embodiment, code (e.g., executable code or source code) is stored in a set of one or more non-transitory computer readable storage media that store executable instructions (or have other memory for storing executable instructions) that, when executed by (i.e., as a result of) one or more processors of a computer system, cause the computer system to perform the operations described herein. In at least one embodiment, the set of non-transitory computer readable storage media comprises a plurality of non-transitory computer readable storage media, where one or more of the individual non-transitory storage media of the plurality of non-transitory computer readable storage media do not contain all the code, but the plurality of non-transitory computer readable storage media collectively store all the code. In at least one embodiment, the executable instructions are executed such that different instructions are executed by different processors.

[0192] Thus, in at least one embodiment, a computer system is configured to implement one or more services that, singly or collectively, perform the operations of the processes described herein, and such computer system is configured with applicable hardware and / or software that enables the operations to be performed. Further, a computer system that implements at least one embodiment of the present disclosure may be a single device, or in another embodiment, a distributed computer system comprising multiple devices that operate in different ways, such that the distributed computer system performs the operations described herein such that no single device performs all the operations.

[0193] The use of any illustrative or exemplary language (e.g., "such as") provided herein is intended only to further clarify embodiments of the disclosure and does not limit the scope of the disclosure unless otherwise stated. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the disclosure.

[0194] All references cited in this specification, including publications, patent applications, and patents, are hereby incorporated by reference to the same extent as if each reference was individually indicated to be incorporated by reference and was set forth in its entirety herein.

[0195] In the specification and claims, the terms "coupled" and "connected," along with their derivatives, may be used. It should be understood that these terms may not be intended as synonyms for each other. Rather, in certain instances, "connected" or "coupled" may be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. "Coupled" may also mean that two or more elements are not in direct contact with each other, but still interlock or interact with each other.

[0196] Unless specifically noted otherwise, terms such as "processing," "computing," "calculating," or "determining" throughout the specification refer to the acts and / or processes of a computer or computing system, or similar electronic computing device, that manipulate and / or transform data represented as physical quantities, such as electronic quantities, in the registers and / or memory of the computing system into other data similarly represented as physical quantities in the memory, registers, or other such information storage, transmission, or display device of the computing system.

[0197] Similarly, the term "processor" may refer to any device, or portion of a device, that processes electronic data from registers and / or memory and converts the electronic data into other electronic data that can be stored in registers and / or memory. A "computing platform" may comprise one or more processors. A "software" process as used herein may include software and / or hardware entities that perform work over time, such as, for example, tasks, threads, and intelligent agents. Also, each process may refer to multiple processes for serially or in parallel, continuously or intermittently executing instructions. In at least one embodiment, the terms "system" and "method" are used interchangeably herein to the extent that a system may embody one or more methods and a method may be considered a system.

[0198] As used herein, reference may be made to obtaining, acquiring, receiving, or inputting analog or digital data into a subsystem, computer system, or computer-implemented machine. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog and digital data may be implemented in a variety of ways, such as receiving the data as a parameter of a function call or a call to an application programming interface. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data may be implemented by transferring the data over a serial or parallel interface. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data may be implemented by transferring the data over a computer network from a providing entity to an acquiring entity. In at least one embodiment, reference may also be made to providing, outputting, transmitting, sending, or presenting analog or digital data. In various examples, the process of providing, outputting, transmitting, sending, or presenting analog or digital data may be accomplished by transferring the data as an input or output parameter of a function call, an application programming interface, or an inter-process communication mechanism.

[0199] Although the description herein sets forth example implementations of the described techniques, other architectures may be used to implement the described functionality, and are intended to be within the scope of this disclosure. Additionally, while a specific distribution of roles is defined above for purposes of illustration, the various functions and roles may be distributed and divided in different manners depending on the circumstances.

[0200] Furthermore, although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter claimed in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claims.

Claims

1. sending a preload request using an application hosting platform to preload a first portion of data for an application hosted using the application hosting platform, the preload request being received prior to receiving user input identifying the application to execute; receiving, using the application hosting platform, a first indication that the first portion of data has been preloaded for the application using a virtualized computing environment; receiving a user request to execute the application using the application hosting platform; sending a load request using the application hosting platform to load a second portion of data for the application into the virtualized computing environment; receiving, using the application hosting platform, a second indication that the second portion of data has been loaded into the virtualized computing environment for the application; in response to receiving the second instruction, executing the application using the virtualized computing environment; A method comprising:

2. The method of claim 1 , wherein the first portion of data is data excluding data related to the user request, and the second portion of data is related to the user request.

3. using the application hosting platform to send a plurality of preload requests to preload the first portion of data for the application in a plurality of virtualized computing environments, the plurality of preload requests being received prior to receiving user input identifying the application to execute; receiving a third indication, using the application hosting platform, that the first portion of data has been preloaded for the application in the plurality of virtualized computing environments; The method of claim 1 further comprising:

4. receiving a second user request to execute the application running in a second virtualized computing environment of the plurality of virtualized computing environments using the application hosting platform; sending a second load request using the application hosting platform to load the second portion of data for the application to the second virtualized computing environment; receiving a fourth indication, using the application hosting platform, that the second portion of data has been loaded for the application into the second virtualized computing environment; in response to receiving the fourth instruction, causing execution of the application in the second virtualized computing environment; The method of claim 3 further comprising:

5. The method of claim 2 , further comprising determining the number of virtualized computing environments to preload based on an average duration that the virtualized computing environments are idle.

6. The method of claim 1 , wherein the virtualized computing environment is placed in an idle state after receiving the indication that the first data has been preloaded.

7. The method of claim 1 , wherein preloading of a game is requested using a command line interface (CLI) command or a flag in a configuration file of the application.

8. 10. The method of claim 1, further comprising: predicting that the first portion of data will be preloaded for the application in one or more virtualized computing environments using a machine learning model trained based at least in part on a previous duration for preloading the first portion of data for the application in the virtualized computing environments.

9. receiving, using the application, an indication that one or more files associated with the application have been updated; in response to receiving the indication that the one or more files associated with the application have been updated, dismantling the virtualized computing environment; The method of claim 1 further comprising:

10. 1. An application hosting platform comprising: requesting allocation of a virtualized computing environment to launch an application hosted on the hosting platform in a pre-loaded mode prior to receiving user input identifying an application to execute, the pre-loaded mode being associated with pre-loading a first portion of data for the application; in response to a first indication that the virtualized computing environment has been allocated for the application, sending instructions to launch the application in the preload mode in the virtualized computing environment; receiving a second indication that the first portion of data has been loaded into the virtualized computing environment; transitioning the virtualized computing environment to an idle state in response to receiving the second instruction. The system includes an application hosting platform for performing the above steps.

11. The application hosting platform further comprises: receiving a request to execute the application in the virtualized computing environment, the request being associated with a user input identifying the application to execute; sending second instructions to load a second portion of data for the application into the virtualized computing environment; and The system of claim 10 , further comprising:

12. The application hosting platform further comprises: receiving a third indication that the second portion of data has been loaded into the virtualized computing environment; and in response to receiving the second instruction, executing the application in the virtualized computing environment. The system of claim 11 , further comprising:

13. The application hosting platform further comprises:

12. The system of claim 11, in response to receiving the request to run the application in the virtualized computing environment, accessing an account associated with the user input, the account including the second portion of data.

14. 11. The system of claim 10, wherein the application is instructed to launch in the preloaded mode using a command line interface (CLI) command or a flag in the application's configuration file.

15. The application hosting platform further comprises: receiving an initialization sequence prior to allocating the virtualized computing environment; receiving, after receiving the initialization sequence, the indication that the first portion of data has been loaded into the virtualized computing environment based on the application running in a cloud environment; The system of claim 10 , further comprising:

16. The application hosting platform further comprises:

11. The system of claim 10, further comprising: determining that a number of user inputs associated with the application is greater than a number of virtualized computing environments in which the application is launched; and allocating the virtualized computing environments based on determining that the number of user inputs is greater than the number of virtualized computing environments.

17. The application, Game applications, Video conferencing applications, Content creation applications, Cloud-hosted applications, Collaborative content creation applications, Cloud-hosted collaborative content creation application, Video streaming applications, Machine learning applications, or Simulation Applications The system of claim 10 , comprising at least one of:

18. The application hosting platform, Cloud-hosted game service platform, A cloud-hosted collaborative content creation platform for heterogeneous content creation applications; Video streaming hosting platform, A testing platform for simulated or augmented content; Machine learning training platform, Machine learning implementation platform, or Video Conferencing Hosting Platform The system of claim 10 , comprising at least one of:

19. A non-transitory computer-readable medium storing instructions that, when executed by a processing device, cause the processing device to: sending a preload request to preload a first portion of data for an application hosted using an application hosting platform, the preload request being received prior to receiving user input identifying the application to execute; receiving, using the application hosting platform, a first indication that the first portion of data has been preloaded for the application using a virtualized computing environment; receiving a user request to execute the application using the application hosting platform; sending a load request to load a second portion of data for the application into the virtualized computing environment using the application hosting platform; receiving a second indication that the second portion of data has been loaded into the virtualized computing environment for the application; in response to receiving the second instruction, executing the application using the virtualized computing environment; and A non-transitory computer-readable medium for causing

20. 20. The non-transitory computer-readable medium of claim 19, wherein the first portion of data is data excluding data related to the user request, and the second portion of data is related to the user request.

21. The instructions, when executed by the processing device, cause the processing device to: sending a plurality of preload requests to preload the first portion of data for the application using a plurality of virtualized computing environments, the plurality of preload requests being received prior to receiving user input identifying the application to execute; receiving a third indication that the first portion of data has been preloaded for the application in the plurality of virtualized computing environments; and 20. The non-transitory computer-readable medium of claim 19, further comprising:

22. The instructions, when executed by the processing device, cause the processing device to: receiving a second user request to execute the application using a second virtualized computing environment of the plurality of virtualized computing environments; sending a second load request to the second virtualized computing environment to load the second portion of data for the application; receiving a fourth indication that the second portion of data has been loaded for the application using the second virtualized computing environment; and in response to receiving the fourth instruction, executing the application using the second virtualized computing environment.

22. The non-transitory computer-readable medium of claim 21, further comprising:

Images