Multi-system AI repository controller

Abstract

The embodiment of the invention relates to a multi-system AI repository controller. Systems and methods are provided for creating a hierarchical repository of machine learning (ML) models in which ML models are stored as open container images (OCIs). The OCI includes a layer or portion of a complete ML model so that the model can be stored separately and as a smaller file. The ML model may be pre-packed for automated downloading and integration at the customer site. In some examples, the OCI may identify / store the model in a directory structure defining the model and its configuration files. The OCI may include a combination of layers and configuration files that allow the AI platform to optimize storage and simplify the process of downloading or updating a given user namespace rather than downloading a single large file for the model.

Description

[0001] Cross-references to related applications

[0002] This application, together with U.S. Patent Application No. 18 / 823,115 (Case No.: P175597US(109793-01455)) and Indian Patent Application No. 202441065736 (Case No.: P175628IN(109793-10456)), are jointly pending and their contents are incorporated herein by reference in their entirety for all purposes. Background Technology

[0003] Machine learning models are the processes that enable computers to learn from data and make decisions or predictions without explicit programming for a specific task. These models improve their performance as they are exposed to more data over time.

[0004] Machine learning models can be categorized into supervised learning models (such as linear regression and classification models), unsupervised learning models (such as clustering and dimensionality reduction), reinforcement learning models, and semi-supervised and self-supervised learning models. One type of machine learning model is the Large Language Model (LLM), which involves both unsupervised and self-supervised learning. In LLMs, these models are specifically designed to understand, generate, and manipulate human language to analyze large amounts of data through advanced neural network architectures and procedural execution.

[0005] Because these LLMs are highly complex, customer environments often run them remotely from the model provider or other remote systems. However, using remote systems can introduce security issues such as transmitting data over open networks, restricting access to sensitive data for authorized users, or sharing remote systems with other entities that may be direct competitors of the customer. Attached Figure Description

[0006] This disclosure is described in detail with reference to the following figures, based on one or more various examples. These figures are provided for illustrative purposes only and depict only typical, non-limiting aspects of such examples.

[0007] Figure 1 The illustration shows an example of a network configuration that can be implemented for organizations such as businesses, educational institutions, government entities, healthcare facilities, or other organizations.

[0008] Figure 2 This is an illustrative AI platform with user namespaces and a set of repositories, as shown in some examples of this disclosure.

[0009] Figure 3 This is an illustrative AI platform with user namespaces and a set of repositories, as shown in some examples of this disclosure.

[0010] Figure 4 The illustrations show some examples of the process for downloading and updating Open Container Images (OCI).

[0011] Figure 5 The illustrations show some examples of Open Container Images (OCI) in this disclosure.

[0012] Figure 6 These are some examples of illustrative interfaces provided by the AI ​​platform in this disclosure.

[0013] Figure 7 These are example computational components that, based on the examples discussed in this article, can be used to implement various features of the model set.

[0014] Figure 8 It is a computing component that can be used to implement examples of the disclosed technology.

[0015] The accompanying drawings are not exhaustive and do not limit this disclosure to the precise form disclosed. Detailed Implementation

[0016] Traditional LLMs are large (e.g., 50GB to 100GB models), and due to network technology limitations, downloads can take days within the customer's computing environment. For example, the customer site may have an intranet, private cloud, or other private network accessible only to authenticated and authorized members of the customer site, while other entities can attempt to communicate with devices at the customer site using the internet or other public networks. This limits the ability of customer sites to leverage machine learning models within private, secure networks.

[0017] The disclosed AI platform examples create a hierarchical repository of machine learning (ML) models, where ML models are stored as Open Container Images (OCIs). OCIs comprise layers or portions of a complete ML model, allowing models to be stored separately as smaller files. ML models can be pre-packaged for automated download and integration at customer sites. In some examples, OCIs can identify / store models within a directory structure that defines the model and its configuration file. OCIs can include combinations of layers and configuration files, allowing the AI ​​platform to optimize storage and streamline the process of downloading or updating a given user namespace instead of downloading a single large file for the model.

[0018] Once the AI ​​platform has generated the OCI, it can provide it to a model hierarchical repository (e.g., a global repository) that synchronizes the OCI with the OCI's public marketplace. The local repository associated with the client's environment can identify the updated OCI on the public marketplace and initiate an automatic download for the OCI (or a portion / layer of the OCI). The AI ​​platform can automatically extract the OCI from the local repository into the model volume while maintaining the same directory structure. In some examples, the model volume can be mounted into the inference service to read the model from the local cache, rather than downloading the model to the client's computing environment during runtime.

[0019] The extraction process can store the model in a local cache, which clones the model / layer to the user namespace. The user namespace can then expose the downloaded model to users operating client devices to utilize it. In some examples, the model can be exposed to users via an interface displayed on the client device, allowing users to select and deploy the model if they have the appropriate authorization to deploy / execute it.

[0020] In response to updates to models / layers, the AI ​​platform can initiate a synchronization process that automatically updates local repositories, caches, and user namespaces. This synchronization process can push any updates to deployed models or other components in the customer's computing environment.

[0021] Throughout the disclosure, technical improvements are illustrated. For example, the use of OCI can include a combination of layers and profiles that allows AI platforms to optimize storage and streamline the process of downloading or updating a given user namespace instead of downloading a single large file for the model. This can create more efficient processing, allowing a larger portion of bandwidth to be reserved for other components of the system.

[0022] Before describing in detail the various examples of the disclosed systems and methods, it is useful to describe an example network installation that can be implemented in various applications using this example network installation. Figure 1 The illustration shows an example of a network configuration 100 that can be implemented for organizations such as businesses, educational institutions, government entities, healthcare facilities, or other organizations. Figure 1The illustration shows an example configuration implemented by an organization with multiple users (or at least multiple client devices 110) and possibly multiple physical or geographic sites 102, 132, 142. Network configuration 100 may include a main site 102 communicating with a network 120 storing the AI ​​platform. Network configuration 100 may also include one or more remote sites 132, 142, each of which may store portions / components of the AI ​​platform. In some examples, the main site 102 and remote sites 132, 142 may also store various registries / repositories and the OCI marketplace.

[0023] Main site 102 may include a main network, which can be, for example, an office network, a home network, or other network installation. The main network may be a private network, such as a network that may include security and access controls to restrict access to authorized users of the private network. Authorized users may include, for example, employees of the company located at main site 102, residents of a residence, customers of the business, etc.

[0024] exist Figure 1 In the example, master site 102 includes controller 104 that communicates with network 120. Controller 104 can provide communication with network 120 to master site 102. In addition to controller 104, other communication points with network 120 may exist for master site 102. Although a single device associated with controller 104 is illustrated, master site 102 may include multiple controllers and / or multiple communication points with network 120. In some examples, controller 104 may communicate with network 120 via a router. In other examples, controller 104 provides router functionality to devices in master site 102. In this specification, the term "tunnel" refers to the encapsulation mode for data transmission between the AP and the controller.

[0025] Controller 104 may be operable to configure and manage network devices, such as those at the main site 102, and may also manage network devices at remote sites 132, 142. Controller 104 may be operable to configure and / or manage switches, routers, access points, and / or client devices connected to the network. Controller 104 itself may be an access point (AP) or provide access point functionality.

[0026] Controller 104 can communicate with one or more switches 108 and / or wireless access points (APs) 106a to 106c. Switches 108 and wireless APs 106a to 106c provide network connectivity to various client devices 110a to 110j. Using the connection to switches 108 or APs 106a to 106c, client devices 110a to 110j can access network resources, including the (main site 102) network and other devices on network 120.

[0027] Examples of client devices may include: desktop computers, laptop computers, servers, web servers, authentication servers, Authentication, Authorization and Accounting (AAA) servers, Domain Name System (DNS) servers, Dynamic Host Configuration Protocol (DHCP) servers, Internet Protocol (IP) servers, Virtual Private Network (VPN) servers, network policy servers, mainframes, tablet computers, e-readers, netbook computers, televisions and similar monitors (e.g., smart TVs), content receivers, set-top boxes, personal digital assistants (PDAs), mobile phones, smartphones, smart terminals, dumb terminals, virtual terminals, video game consoles, virtual assistants, Internet of Things (IoT) devices, etc.

[0028] Within the main site 102, a switch 108 is included as an example of an access point for the network established in the main site 102 for wired client devices 110i to 110j. Client devices 110i to 110j can connect to switch 108 and, through switch 108, can access other devices within network configuration 100. Client devices 110i to 110j can also access network 120 through switch 108. Client devices 110i to 110j can communicate with switch 108 via wired or wireless connection 112. In the illustrated example, switch 108 communicates with controller 104 via wired or wireless connection 112.

[0029] Wireless APs 106a to 106c are included as another example of access points for the network established in main site 102 for client devices 110a to 110h. Each AP in APs 106a to 106c can be a combination of hardware, software, and / or firmware configured to provide wireless network connectivity to wireless client devices 110a to 110h. Figure 1 In the example, APs 106a to 106c can be managed and configured by controller 104. APs 106a to 106c communicate with controller 104 and the network via connection 112, which can be a wired or wireless interface.

[0030] Network configuration 100 may include one or more remote sites 132. Remote sites 132 may be located in a different physical or geographical location than the main site 102. In some cases, remote sites 132 may be located in the same geographical location as the main site 102, or may be in the same building, but lack a direct connection to the network located within the main site 102. Instead, remote sites 132 may utilize connections on a different network (e.g., network 120). Remote sites 132 (such as...) Figure 1A remote site 132 illustrated may be, for example, a satellite office, another floor in a building, or a suite. Remote site 132 may include a gateway device 134 for communicating with network 120. Gateway device 134 may be a router, a digital-to-analog modem, a cable modem, a digital subscriber line (DSL) modem, or some other network device configured to communicate with network 120. Remote site 132 may also include a switch 138 and / or an access point (AP) 136 communicating with gateway device 134 via a wired or wireless connection. Switch 138 and AP 136 provide connectivity to the network to various client devices 140a to 140d.

[0031] In various examples, remote site 132 can communicate directly with master site 102, enabling client devices 140a to 140d at remote site 132 to access network resources at master site 102 as if these client devices 140a to 140d were located at master site 102. In this example, remote site 132 is managed by controller 104 at master site 102, and controller 104 provides the necessary connectivity, security, and accessibility to establish a connection between remote site 132 and master site 102. Once connected to master site 102, remote site 132 can be used as part of a private network provided by master site 102.

[0032] In various examples, network configuration 100 may include one or more smaller remote sites 142, which consist only of a gateway device 144 and a wireless access point 146 for communicating with network 120, through which various client devices 150a to 150b access network 120. An example of remote site 142 could represent, for example, a single employee's home or a temporary remote office. Remote site 142 may also communicate with master site 102, enabling client devices 150a to 150b at remote site 142 to access network resources at master site 102 as if they were located at master site 102. Remote site 142 may be managed by controller 104 at master site 102 to enable this transparency. Once connected to master site 102, remote site 142 can be used as part of a private network provided by master site 102.

[0033] Network 120 may be a public or private network (such as the Internet) or other communication network to allow connectivity between various sites 102, 132 to 142 and access to servers 160a to 160b. Network 120 may include third-party telecommunications lines, such as telephone lines, broadcast coaxial cables, fiber optic cables, satellite communications, cellular communications, etc. Network 120 may include any number of intermediate network devices, such as switches, routers, gateways, servers, and / or controllers, which are not directly part of network configuration 100 but facilitate communication between the various parts of network configuration 100 and between network configuration 100 and other network-connected entities. Network 120 may include various servers 160a to 160b. In the example, servers 160a to 160b may include content servers that include various providers of downloadable and / or streaming multimedia content, including audio, video, graphics, and / or text content or any combination thereof. Examples of content servers 160a to 160b include web servers, streaming radio and video providers, and cable and satellite television providers. Client devices 110a to 110j, 140a to 140d, and 150a to 150b can request and access multimedia content provided by content servers 160a to 160b.

[0034] In another example, servers 160a to 160b may include a flow optimization service server that includes various information for providing services and optimizing traffic flows to client devices 110a to 110j, client devices 140a to 140d, and client devices 150a to 150b, according to the examples disclosed herein. Access points 106a to c, 136, and 146; switch 108; and gateway devices 134 and 144 may request or upload information, such as telemetry data, for optimizing the presentation of services to client devices 110a to 110j, client devices 140a to 140d, and client devices 150a to 150b. This information may include, but is not limited to, QoE metrics or estimates based on each traffic flow (e.g., referred to herein as QoE scores); flow characteristics and other QoS measurements, such as, but not limited to, jitter, latency, call duration, delay, etc.; analytics; transport protocols (e.g., OFDMA and MU-MIMO), etc. This information can be stored in a database that can be communicatively coupled to servers 160a and 160b. In the example, servers 160a to 160b can be cloud-based, which, as understood by those skilled in the art, means remotely hosted (rather than hosted on a local server / computer) on a network and is remotely accessible.

[0035] In the cloud-based example of servers 160a to 160b, servers 160a to 160b can store components for both the public and private clouds, respectively. The private cloud can implement an integrated AI platform at the customer's site to deploy ML models from a model repository. In some examples, the server implementing the public cloud can store machine learning models that can be downloaded to and integrated into the private cloud.

[0036] Figure 2 This disclosure presents illustrative AI platforms with user namespaces and repository collections as examples. In example 200, AI platform 202, communication structure 220, dev ops process 230, repository 240, marketplace 250, and NGC 260 are shown.

[0037] AI Platform 202 (e.g., AI Essentials) TM This is configured to synchronize and update layers of OCI and / or the model across the entire platform and external devices. In some examples, OCI is stored as Docker container layers. These layers can be split or separated so that when one layer is changed / updated, the entire image does not need to be changed / updated. In this case, any updated layer can be stored or sent separately from the other layers of OCI. In these examples, a layer or portion of OCI corresponds to a part of the complete ML model, so the model can be stored separately as smaller files.

[0038] AI platform 202 includes user namespace 204 (illustrated as first user namespace 204A and second user namespace 204B) and AI system 210. User namespace 204 is configured to partition cluster resources (e.g., in Kubernetes) across multiple users or applications. TM (Within the platform). In some examples, the user namespace 204 can act as a whole for Kubernetes. TM A virtual cluster within a cluster or AI platform 202. In some examples, user namespace 204 can isolate resources in a first user namespace from resources in other user namespaces (e.g., namespace-specific resources are separated from each other). This separation helps the AI ​​platform 202 manage resource quotas and limits and reduces the risk of naming conflicts. In some examples, user namespace 204 can help implement different access controls and policies for different parts of the cluster. Access control can implement role-based access control (RBAC) or token-based access control to grant namespace-specific permissions.

[0039] User namespace 204 may include inference service 206 (illustrated as first inference service 206A and second inference service 206B) and model repository 208 (illustrated as first model repository 208A and second model repository 208B). Inference service 206 is configured to automatically determine the appropriate namespace for deploying resources based on parameters such as the type of application or its environment (e.g., development, tier, production)

[0040] Model repository 208 can correspond to a persistent data repository that restricts access to data used for a specific namespace. Each model repository 208 can restrict access to data from other namespaces.

[0041] AI systems 210 (e.g., EZAI-systems) TM The AI ​​platform 202 is configured to define a control plane that uses a local cache 212, AI operators 214, an AI repository / registry controller 216, and a model downloader process 218 to monitor, deploy, and update services. For example, the local cache 212 may include data that has already been copied from a local repository 224 at the communication structure 220. Once data is stored in the local cache 212, the AI ​​platform 202 is configured to clone the data to the appropriate user namespace 204 (e.g., based on policy or other determinations).

[0042] AI Operator 214 is configured to deploy and manage applications using custom controllers and resources. In some examples, AI Operator 214 deploys AI Repository / Registry Controller 216 to manage the application's lifecycle based on the state defined in a Custom Resource Definition (CRD). The controller can perform complex tasks such as application scaling, backup, and updates.

[0043] The model downloader process 218 is configured to automatically download OCI from the local repository 224. The model downloader process 218 is also configured to extract machine learning models from the local repository 224 (e.g., layers of OCI) to the local cache 212.

[0044] AI system 210 is also configured to clone the OCI layer from local cache 212 to user namespace 204, where machine learning models can be used in the inference process. The model can be installed into inference service 206 to read the model from local cache 212 or model store 208 (namespace local), instead of downloading the model during runtime.

[0045] Communication structure 220 (e.g., EZFab OVA) may correspond to a virtual machine (VM) image file format used to store data in a virtualization environment. Communication structure 220 includes data that has been pushed / downloaded by dev ops process 230 and stored using local storage library 224.

[0046] Dev ops process 230 is configured to coordinate the downloading and storage of data across the network, including data stored using global repository 240 and NGC directory 260, and to provide / push data to marketplace 250. In some examples, dev ops process 230 executes a set of processes / jobs, including conversion to OCI image process 232 and download to local path process 234.

[0047] The conversion to an OCI image process 232 is configured to generate an Open Container Image (OCI) that includes a collection of layers of a machine learning model. For example, process 232 may pull / download the ML model from NGC 260 and provide the OCI to a global repository 240.

[0048] In some examples, the download to local path process 234 identifies the source (e.g., NGC directory 260) and destination (e.g., local repository 224 or repository 240) of the data. In some examples, repository 240 can synchronize OCI with a public marketplace 250 that includes multiple Open Container Images (OCIs). In other examples, AI platform 202 can automatically download the OCI layer from global repository 240 to local repository 224 in a customer's computing environment.

[0049] Storage 240 is configured to act as a global storage for communicating with marketplace 250 and communication structure 220. Storage 240 can store OCI pushed from dev ops process 230. Storage 240 can also synchronize OCI with marketplace 250.

[0050] Marketplace 250 is configured as a tier for storing OCI (e.g., within a container repository). In some examples, Marketplace 250 is an online platform that provides access to download and manage data, purchase hardware or software, and access to applications / products and services.

[0051] NGC directory 260 (such as the NVIDIA GPU Cloud (NGC) directory) is configured as a repository of pre-built software containers and other resources, which helps simplify application and workflow deployment. For example, NGC directory 260 can include Docker containers that contain machine learning models such as TensorFlow, PyTorch, and MXNet. In some examples, containers are pre-configured with various protocols (such as NVIDIA's CUDA and cuDNN libraries) to help speed up communication with the containers.

[0052] In some examples, NGC Catalog 260 includes development tools and libraries that facilitate the development and deployment of GPU-accelerated applications. As illustrative examples, these tools and libraries may include NVIDIA's RAPIDS Data Science Library, NVIDIA Data Loading Library (DALI), and other utilities designed to enhance performance and ease of use.

[0053] In some examples, NGC catalog 260 implements version control for containers and models. Version control helps users access and deploy specific versions of applications, data, etc.

[0054] In the illustrative example, Figure 2 The illustrated system can be implemented for classifying ML models. Model classification by the AI ​​platform 202 may involve two phases. The first phase may include downloading ML models from the NGC catalog 260. These can be converted into OCI images via process 232 and pushed to storage 240. Storage 240 is identified when a product package is created. The second phase may include deploying and managing OCI using AI operators 214. To store and render models, models can be pulled from local storage 224 and extracted to local cache 212.

[0055] Figure 3 This disclosure presents illustrative AI platforms with user namespaces and collections of repositories as examples. In example 300, AI platform 302, dev ops process 304, DSCC / GLP 310, communication structure agent 320, upgrade controller 330, AI controller 340, local repository 348, marketplace 350, and NGC 360 are shown. Figure 3 The AI ​​platform 302, DevOps process 304, local repository 348, marketplace 350, and NGC catalog 360 shown in the diagram can respectively correspond to Figure 2 The diagram shows the AI ​​platform 202, dev ops process 230, local repository 224, marketplace 250, and NGC catalog 260.

[0056] Dev ops process 304 is configured to monitor updates to NGC catalog 360 and coordinate the download and storage of data across the network after the data has been downloaded. In some examples, dev ops process 304 uses data downloaded from NGC catalog 360 to generate OCI. Dev ops process 304 is also configured to provide OCI to the corresponding mirror repository accessible to marketplace 350 or AI platform 302. Once the OCI is uploaded / pushed to marketplace 350, the AI ​​platform's local repository 348 can receive the OCI from marketplace 350.

[0057] AI platform 302 includes a communication infrastructure agent 320, an upgrade controller 330, an AI controller 340, and a local storage 348. The communication infrastructure agent 320 is configured to receive triggered updates from DSCC / GLP 310 and synchronize multiple layers of OCI 322 with various storage repositories, including the local storage 348. In some examples, DSCC / GLP 310 is configured as a data repository whose updates are monitored by the communication infrastructure agent 320.

[0058] In some examples, the communication structure agent 320 can also trigger the upgrade controller 330 of the AI ​​platform 302 to indicate that data has changed. In this example, the communication structure agent 320 is configured to implement the version control process of the OCI layer. When the upgrade controller 330 is triggered, a download process can be initiated to access the local repository and convert the new files into the OCI directory structure. Figure 5 (Further illustration is provided). The inference service in the user namespace can read updated files from the local cache populated by the controller.

[0059] AI controller 340 is configured to operate as a control loop that monitors the state of the cluster in AI platform 302 and requests changes to the cluster. For example, when the current state deviates from the desired state, AI controller 340 can take action to reconcile the discrepancies. As an illustrative example, if a deployment specifies three replicas but only two are running, the deployment controller can spawn additional Pods to match the desired state.

[0060] AI controller 340 includes a set of processes / jobs, including extracting OCI format 342, data repository 344, and recreating instances 346. Extracting OCI format 342 is configured to automatically download OCI from local repository 348. Extracting OCI format 342 is also configured to extract machine learning models to data repository 344 (e.g., as a local cache) and recreate any instances of NIM, including service container images and models.

[0061] Figure 4 The illustrations depict processes for downloading and updating Open Container Images (OCI) in some examples of this disclosure. Example 400 shows a dev ops process 410, a marketplace system 420, a communication structure 430, and an AI platform 440. Figure 4 The DevOps process 410, market system 420, communication structure 430, and AI platform 440 can be respectively similar to Figure 2 The dev ops process 230, market 250, communication structure 220, and AI platform 202 are included.

[0062] In box 450, the dev ops process 410 generates an OCI and provides it to the marketplace system 420. The OCI comprises layers or parts of the complete ML model, allowing the model to be stored separately as smaller files.

[0063] In box 452, market system 420 synchronizes OCIs with a global repository. For example, once dev ops process 410 has generated an OCI, dev ops process 410 can provide the OCI to a global repository that synchronizes the OCI with market system 420, which stores a collection of OCIs from multiple platforms.

[0064] In block 454, communication structure 430 can download at least one layer of OCI from market system 420 to local storage. For example, a local storage associated with a customer environment can identify the updated OCI on market system 420 and initiate an automatic download for OCI (or a portion / layer of OCI).

[0065] In box 456, AI platform 440 extracts layers from OCIs from the local repository to the local cache. For example, AI platform 440 can automatically extract OCIs from the local repository to the model volume while maintaining the same directory structure.

[0066] In box 460, AI platform 440 clones the layer from OCI to the user namespace. For example, the extraction process may store the model in a local cache, which then clones the model / layer to the user namespace. The user namespace can expose the downloaded model to users operating client devices to utilize it. In some examples, the model may be exposed to users via an interface displayed on the client device, allowing users to select and deploy the model if they have the appropriate authorization to deploy / execute it.

[0067] In some examples, model volumes can be mounted into the inference service of a user namespace to read the model from a local cache, rather than downloading the model to the client's computing environment during runtime.

[0068] In some examples, the download / cloning process from the local cache to the user namespace can simplify the process of downloading or updating a given user namespace instead of downloading a single large file for the model.

[0069] In box 462, AI platform 440 updates its local cache and synchronizes it with local storage and communication infrastructure 430. For example, in response to an update to a model / layer, AI platform 440 can initiate a synchronization process that automatically updates the local storage, cache, and user namespace. The synchronization process can push any updates to deployed models or other components in the customer's computing environment.

[0070] In some examples, in response to an update to the OCI layer, the AI ​​platform 440 can initiate a synchronization process that automatically updates the local repository, cache, and user namespace of the client's computing environment.

[0071] Figure 5 The illustrations depict Open Container Images (OCIs) in some examples of this disclosure. In Example 500, the components of an OCI for an ML model stored in a NIM are illustrated. In some examples, the OCI may identify / store the model within a directory structure that defines the model and its configuration files. A NIM may include two parts: a service container image and the model, and each model may have one or more configuration files.

[0072] Example 500 illustrates four profiles for a snapshot of model 502. In this example, ML model 502 consists of several profiles (boxes 504, 520, 540, and 560), and each profile consists of several blobs (506, 508, 510, 512, 514, 522, 524, 526, 528, 530, 562, 564, 566, 568, 570, 572, 574, 576, 578, and 580). The AI ​​platform can create an OCI image (boxes 504, 520, 540, and 560) for each profile, and this image can consist of several layers corresponding to the blobs. This format allows the system to optimize storage and simplifies the process of downloading or updating a given profile instead of downloading a large single file for the model.

[0073] In some examples, the model's layers (boxes 566, 568, 570, 572) are separated into files smaller than the entire model. For example, these layers could each correspond to a 20GB file, and the entire directory size could correspond to an 80GB file. For instance, when an update is implemented on a security tensor within a security tensor in the model layers, a single layer can be updated, and the remaining layers can remain unchanged.

[0074] In some examples, the standard model repository format is identified. The standard model repository can be followed by other repository systems (such as Facets and NGC directories). The standard model repository can be converted to OCI layers, which are stored in the local and global repositories described herein.

[0075] Figure 6These are illustrative interfaces provided by an AI platform, as shown in some examples of this disclosure. In example 600, the interface may illustrate multiple models 610 (illustrated as first model 610A, second model 610B, third model 610C, and fourth model 610D). Each model may be illustrated with a model name to identify the ML model used by the user to deploy in the customer's environment. When the user selects a model, the interaction may highlight the model on the interface, as shown in highlighted model 602, triggering the download of the ML model to a local cache cloned to the user's namespace.

[0076] It should be noted that the terms “optimization,” “optimal,” etc., used in this document can be used to mean manufacturing or achieving performance as efficiently or perfectly as possible. However, as a person skilled in the art will recognize upon reading this document, perfection cannot always be achieved. Therefore, these terms can also cover manufacturing or achieving performance as well or practically as possible, or manufacturing or achieving better performance than could be achieved using other settings or parameters, given a given condition.

[0077] Figure 7 The illustrations depict computational components that can be used to implement genealogy-based network event classification, according to various examples of the disclosed techniques. Now refer to... Figure 7 The computing component 700 can be, for example, a server computer, a controller, or any other similar computing component capable of processing data. Figure 7 In the example implementation, computing component 700 includes hardware processor 702 and machine-readable storage medium 704.

[0078] The hardware processor 702 may be one or more central processing units (CPUs), semiconductor-based microprocessors, and / or other hardware devices suitable for retrieving and executing instructions stored in the machine-readable storage medium 704. The hardware processor 702 may fetch, decode, and execute instructions, such as instructions 706 to 712, to control processes or operations for genealogy-based network event classification. Alternatively, or in addition to retrieving and executing instructions, the hardware processor 702 may include one or more electronic circuits comprising electronic components for the function of executing one or more instructions, such as field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), or other electronic circuits.

[0079] Machine-readable storage media (such as machine-readable storage media 704) can be any electronic, magnetic, optical, or other physical storage device that contains or stores executable instructions. Therefore, machine-readable storage media 704 can be, for example, random access memory (RAM), non-volatile RAM (NVRAM), electrically erasable programmable read-only memory (EEPROM), storage devices, optical discs, etc. In some examples, machine-readable storage media 704 can be a non-transient storage medium, where the term "non-transient" does not cover transient propagation signals. As described in detail below, machine-readable storage media 704 can be encoded with executable instructions, such as instructions 706 to 712.

[0080] The hardware processor 702 can execute instructions 706 to automatically download the layers of an Open Container Image (OCI) from a global repository to a local repository in the client's computing environment. In some examples, the OCI comprises a set of layers of a machine learning model, which can then be stored separately as smaller files.

[0081] In some examples, ML models can be pre-packaged for automated download and integration at customer sites. In other examples, OCI can identify / store models within a directory structure that defines the model and its configuration files. OCI can include a combination of layers and configuration files that allows AI platforms to optimize storage and simplify the process of downloading or updating a given user namespace instead of downloading a single large file for the model.

[0082] In some examples, an OCI (Optical Character Set) is generated, comprising a set of layers representing a machine learning model. The generated OCI can be provided to a global repository that synchronizes the OCI with a public marketplace for OCIs. In other examples, once the AI ​​platform has generated the OCI, it can provide it to a model hierarchy repository (e.g., a global repository) that synchronizes the OCI with a public marketplace for OCIs.

[0083] Hardware processor 702 can execute instructions 708 to automatically fetch machine learning models from OCI layers into a local cache in the client's computing environment. In some examples, the local repository associated with the client environment can identify the updated OCI on a public marketplace and initiate an automatic download for the OCI (or a portion / layer of the OCI). The AI ​​platform can automatically fetch the OCI from the local repository into a model volume while maintaining the same directory structure. In some examples, the model volume can be mounted into the inference service to read the model from the local cache, rather than downloading the model to the client's computing environment during runtime.

[0084] Hardware processor 702 can execute instruction 710 to clone the OCI layer from a local cache to a user namespace that utilizes the machine learning model. In some examples, the AI ​​platform can store the model in a local cache, which then clones the model / layer to the user namespace. The user namespace can expose the downloaded model to users operating client devices to utilize it. In some examples, the model can be exposed to users via an interface displayed on the client device, allowing users to select and deploy the model if they have the appropriate authorization to deploy / execute it.

[0085] Hardware processor 702 can execute instructions 712 to initiate a synchronization process that automatically updates the local repository and user namespace of the client computing environment. In some examples, the synchronization process can be initiated in response to a layer being updated. In some examples, the synchronization process can push any updates to deployed models or other components of the client computing environment.

[0086] Figure 8 A block diagram of an example computer system 800 is depicted, in which various examples of the disclosed techniques described herein can be implemented, including the AI ​​platform and other components described herein. The computer system 800 includes a bus 802 or other communication mechanism for communicating information, and one or more hardware processors 804 coupled to the bus 802 for processing information. The hardware processors 804 may, for example, be one or more general-purpose microprocessors.

[0087] Computer system 800 also includes main memory 806, such as random access memory (RAM), cache, and / or other dynamic storage devices, coupled to bus 802, for storing information and instructions to be executed by processor 804. Main memory 806 can also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 804. When stored in a storage medium accessible to processor 804, such instructions enable computer system 800 to become a special-purpose machine customized to perform the operations specified in the instructions.

[0088] The computer system 800 also includes a read-only memory (ROM) 808 or other static storage device coupled to the bus 802 for storing static information and instructions for the processor 804. A storage device 810, such as a disk, optical disk, or USB thumb drive (flash drive), is provided and coupled to the bus 802 for storing information and instructions.

[0089] Computer system 800 may be coupled to display 812, such as a liquid crystal display (LCD) (or touchscreen), via bus 802 for displaying information to a computer user. The display may provide illustrations of optional models, serving as a vending machine with available ML models. The user may interact with the model's tiles. Based on the procedures described throughout this disclosure, the displayed data may be limited to data that the user is authorized to access.

[0090] Computer system 800 may include a user interface module that implements a GUI to be provided to display 812. The user interface module may be stored as executable software code executed by (or more) computing devices in a mass storage device. By way of example, this module and other modules may include components such as software components, object-oriented software components, class components and task components, procedures, functions, properties, programs, subroutines, program code segments, drivers, firmware, microcode, circuit systems, data, databases, data structures, tables, arrays, and variables.

[0091] Generally, as used herein, terms such as “component,” “engine,” “system,” “database,” and “data repository” can refer to logic implemented in hardware or firmware, or to a set of software instructions that may have entry and exit points and are written in programming languages ​​such as, for example, Java, C, or C++. Software modules can be compiled and linked into an executable program, installed in a dynamic link library, or written in an interpreted programming language such as, for example, BASIC, Perl, or Python. It will be understood that software components can be called from other components or from themselves, and / or can be invoked in response to detected events or interrupts. Software components configured to execute on a computing device can be provided on computer-readable media, such as compressed disks, digital video disks, flash drives, magnetic disks, or any other tangible media, or provided as digital downloads (and may initially be stored in a compressed or installable format that requires installation, decompression, or decryption prior to execution). Such software code can be stored, partially or entirely, on a memory device executing the computing device for execution by the computing device. Software instructions can be embedded in firmware, such as EPROM. It will be further understood that the hardware components may include connected logic units, such as gates and flip-flops, and / or may include programmable units, such as programmable gate arrays or processors.

[0092] Computer system 800 may implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic, which, when combined with the computer system, enables computer system 800 to become a special-purpose machine or to program computer system 800 as a special-purpose machine. According to one example of the disclosed techniques, the techniques herein are executed by computer system 800 in response to processor(s) 804 executing one or more sequences of one or more instructions contained in main memory 806. Such instructions may be read into main memory 806 from another storage medium, such as storage device 810. Execution of the instruction sequence contained in main memory 806 causes processor(s) 804 to perform the process steps described herein. In alternative examples, hardwired circuitry may be used in place of or in combination with software instructions.

[0093] As used herein, the term "non-transient medium" and similar terms refer to any medium that stores data and / or instructions that cause a machine to operate in a specific manner. Such non-transient media can include non-volatile media and / or volatile media. Non-volatile media include, for example, optical discs or magnetic disks, such as storage device 810. Volatile media include dynamic memory, such as main memory 806. Common forms of non-transient media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tape or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, PROMs and EPROMs, flash memory EPROMs, NVRAMs, any other memory chips or cartridges, and their networking versions.

[0094] Non-transient media differ from transmission media, but can be used in conjunction with them. Transmission media participate in the transfer of information between non-transient media. For example, transmission media include coaxial cables, copper wires, and optical fibers, including the wires that form bus 802. Transmission media can also take the form of sound waves or light waves, such as those generated during radio wave and infrared data communication.

[0095] Computer system 800 also includes an interface 818 coupled to bus 802. Interface 818 provides bidirectional data communication coupling with one or more network links connected to one or more local networks. For example, interface 818 may be an Integrated Services Digital Network (ISDN) card, a cable modem, a satellite modem, or a modem for providing data communication connectivity with a corresponding type of telephone line. As another example, interface 818 may be a Local Area Network (LAN) card to provide data communication connectivity with a compatible LAN (or a WAN component communicating with a WAN). Wireless links may also be implemented. In any such implementation, interface 818 transmits and receives electrical, electromagnetic, or optical signals, carrying digital data streams representing various types of information.

[0096] Network links typically provide data communication to other data devices over one or more networks. For example, a network link can provide connectivity to a host computer or data equipment operated by an Internet Service Provider (ISP) via a local network. The ISP, in turn, provides data communication services through a global packet data communication network now commonly referred to as the "Internet." Both local networks and the Internet use electrical, electromagnetic, or optical signals that carry digital data streams. Signals over various networks, as well as signals on network links and through interface 818 (carrying digital data to and from computer system 800), are examples of transmission media.

[0097] Computer system 800 can send messages and receive data including program code via (multiple) networks, network links, and interface 818. In the Internet example, the server can send requested code for an application via the Internet, an ISP, a local network, and interface 818.

[0098] The received code may be executed by processor 804 upon receipt and / or stored in storage device 810 or other non-volatile storage device for later execution.

[0099] Each of the processes, methods, and algorithms described in the preceding sections can be implemented in code components executed by one or more computer systems or computer processors including computer hardware, and can be fully or partially automated by them. One or more computer systems or computer processors can also operate to support the execution of related operations in a “cloud computing” environment or as “Software as a Service” (SaaS). Processes and algorithms can be implemented partially or wholly in application-specific circuit systems. The various features and processes described above can be used independently of each other or can be combined in various ways. Different combinations and sub-combinations are intended to fall within the scope of this disclosure, and certain method or process boxes may be omitted in some implementations. The methods and processes described herein are not limited to any particular sequence, and the boxes or states associated with them can be executed in other suitable sequences, or can be executed in parallel, or in some other way. Boxes or states can be added to or removed from the disclosed examples. The execution of certain operations or processes can be distributed among computer systems or computer processors, residing not only within a single machine but also deployed across multiple machines.

[0100] As used herein, the circuits can be implemented using any form of hardware, software, or a combination thereof. For example, one or more processors, controllers, ASICs, PLAs, PALs, CPLDs, FPGAs, logic components, software routines, or other mechanisms can be implemented to form the circuit. In implementation, the various circuits described herein can be implemented as discrete circuits, or the described functions and features can be shared partially or wholly among one or more circuits. Although various features or functional elements can be described or declared separately as separate circuits, these features and functions can be shared among one or more common circuits, and such description should not require or imply that separate circuits are required to implement such features or functions. Where the circuit is implemented wholly or partially using software, such software can be implemented to operate in conjunction with a computing or processing system capable of performing the functions described herein, such as computer system 800.

[0101] As used herein, the term “or” can be interpreted in an inclusive or exclusive sense. Furthermore, descriptions of resources, operations, or structures in the singular form should not be construed as excluding the plural form. Unless otherwise explicitly stated or otherwise understood within the context in which they are used, conditional language (such as “can,” “may,” “possibly,” or “may”) is generally intended to convey that certain features, elements, and / or steps are included in some examples but not in others.

[0102] Unless otherwise expressly stated, the terms and phrases used in this document, and their variations, should be interpreted as open-ended rather than restrictive. Adjectives such as “regular,” “traditional,” “normal,” “standard,” “known,” and similar terms should not be interpreted as limiting the described items to a given time period or to items available at a given time, but rather should be understood to encompass regular, traditional, normal, or standard techniques that may be available or known now or in the future. The presence of extended words and phrases (such as “one or more,” “at least,” “but not limited to,” or other similar phrases in some instances) should not be interpreted as implying a narrower intent or requirement in instances where such extended phrases may not exist.

Claims

1. A computer-implemented method comprising: automatically downloading a layer of an open container image (OCI) from a global repository to a local repository of a customer computing environment, wherein the OCI comprises a set of layers of a machine learning model; automatically extracting the machine learning model from the layer of the OCI to a local cache in the customer computing environment; cloning the layer of the OCI from the local cache to a user namespace that utilizes the machine learning model; and in response to the layer being updated, initiating a synchronization process that automatically updates the local repository and the user namespace of the customer computing environment.

2. The computer-implemented method of claim 1, wherein the OCI is uploaded to the global repository that synchronizes the OCI with a public marketplace of open container images (OCIs).

3. The computer-implemented method of claim 2, wherein the local repository is stored in a private cloud of the customer computing environment that is separate from the public marketplace.

4. The computer-implemented method of claim 1, wherein an operator function performs a job that downloads the layer to the local repository and extracts the layer while maintaining a same directory structure.

6. The computer-implemented method of claim 1, wherein the cloning of the layer of the OCI is implemented by an inference service of the user namespace.

5. The computer-implemented method of claim 4, wherein the operator function is a Kubernetes TM operator, and the same directory structure is a Kubernetes TM cluster directory structure.

7. The computer-implemented method of claim 1, wherein the OCI corresponds to a NVIDIA inference microservice (NIM) that has a service container image and the machine learning model.

8. The computer-implemented method of claim 1, further comprising: providing a reference to a second OCI at an interface; and in response to an interaction received via the interface, initiating the automatic download of the second OCI to the local repository of the customer computing environment.

9. The computer-implemented method of claim 1, wherein the local repository is located in a cluster data structure of a private cloud.

10. The computer-implemented method of claim 1, wherein the update to the layer is an upgrade or a patch to the machine learning model.

11. A private cloud platform comprising: a memory that stores instructions; and a processor that is communicatively coupled to the memory and configured to execute the instructions to: automatically download a layer of an open container image (OCI) from a global repository to a local repository of a customer computing environment, wherein the OCI comprises a set of layers of a machine learning model; automatically extract the machine learning model from the layer of the OCI to a local cache in the customer computing environment; clone the layer of the OCI from the local cache to a user namespace that utilizes the machine learning model; and in response to the layer being updated, initiate a synchronization process that automatically updates the local repository and the user namespace of the customer computing environment. ​ ​ ​ ​ 12. The private cloud platform of claim 11, wherein the OCI is uploaded to the global repository that synchronizes the OCI with a public marketplace of open container images OCIs.

13. The private cloud platform of claim 12, wherein the local repository is stored in a private cloud of the customer computing environment that is separate from the public marketplace.

14. The private cloud platform of claim 11, wherein an operator function performs a job that downloads the layers to the local repository and extracts the layers while maintaining the same directory structure.

15. The private cloud platform of claim 14, wherein the operator function is Kubernetes TM and the same directory structure is a Kubernetes TM cluster directory structure.

16. The private cloud platform of claim 11, wherein the cloning of the layers of the OCI is implemented by an inference service of the user namespace.

17. The private cloud platform of claim 11, wherein the OCI corresponds to a Nvidia inference microservice NIM with a service container image and the machine learning model.

18. The private cloud platform of claim 11, wherein the processor is further configured to: provide, at an interface, a reference to a second OCI; and initiate, in response to an interaction received via the interface, the automatic download of the second OCI to the local repository of the customer computing environment.

19. The private cloud platform of claim 11, wherein the local repository is located in a cluster data structure of a private cloud.

20. A non-transitory computer-readable storage medium storing a plurality of instructions executable by a processor, the plurality of instructions, when executed by the processor, cause the processor to: automatically download, from a global repository, layers of an open container image OCI to a local repository of a customer computing environment, wherein the OCI comprises a set of layers of a machine learning model; automatically extract, from the layers of the OCI, the machine learning model to a local cache in the customer computing environment; clone, from the local cache, the layers of the OCI to a user namespace that utilizes the machine learning model; and in response to the layers being updated, initiate a synchronization process that automatically updates the local repository and the user namespace of the customer computing environment.

Images